Establishing Data Formats and Conventions

I have just released the first Pipe Syntax best practices and style guide! 🐣 As GoogleSQL's Pipe Syntax gains traction, it is important to establish clear conventions for maintainable and efficient code. This guide aims to do just that with: ∙ 𝐂𝐨𝐫𝐞 𝐩𝐫𝐢𝐧𝐜𝐢𝐩𝐥𝐞𝐬: data flow programming, micro-transformations, separation of concerns, and immutable data philosophy. ∙ 𝐂𝐨𝐝𝐞 𝐬𝐭𝐲𝐥𝐞 & 𝐫𝐞𝐚𝐝𝐚𝐛𝐢𝐥𝐢𝐭𝐲: vertical alignment, visual pipeline chunking, argument indentation, and consistent spacing. ∙ 𝐎𝐩𝐞𝐫𝐚𝐭𝐨𝐫-𝐬𝐩𝐞𝐜𝐢𝐟𝐢𝐜 𝐛𝐞𝐬𝐭 𝐩𝐫𝐚𝐜𝐭𝐢𝐜𝐞𝐬: strategic AS usage, AGGREGATE ordering, sequential TVF calls, and efficient filter/join placement. ∙ 𝐀𝐝𝐯𝐚𝐧𝐜𝐞𝐝 𝐭𝐞𝐜𝐡𝐧𝐢𝐪𝐮𝐞𝐬: hybrid queries, production output clarity, templated pipes, and view creation for reusability. ∙ 𝐎𝐭𝐡𝐞𝐫 𝐜𝐨𝐧𝐬𝐢𝐝𝐞𝐫𝐚𝐭𝐢𝐨𝐧𝐬: adherence to standard SQL naming conventions, and preparing for future linter rules. I hope this guide will promote collaboration, reduce onboarding time, and enhance code reusability. 😊 👇 This link is in description 💡 It is still a living document, so I am actively soliciting and incorporating user feedback to ensure the guide remains relevant and useful. Feel free to give me your insights!

Want to grow in Data? Mastering File Formats is non-negotiable. Most beginners focus only on tools - SQL, Python, Spark. But real-world data work is 80% understanding how data is stored, shared, compressed, and moved. Here is a simple breakdown of the file formats every data professional must know - 1. Core Data Formats These are the foundations of almost every pipeline: CSV, JSON, Parquet, Avro - you’ll use them for storage, APIs, and analytics. 2. Big Data & Analytics Formats For large-scale workloads: ORC, Delta, HDF5, Feather/Arrow - built for speed, compression & massive datasets. 3. ML & Deep Learning Formats For training models and storing complex data: TFRecord, Pickle, Video/Audio formats, MAT files - essential for multimodal and scientific work. 4. Geospatial & Scientific Formats Used in climate, maps, and research: GeoJSON, NetCDF, Shapefiles - powerful for specialized domains. 5. System & Infra Formats For configs, backups, and microservices: YAML, Protocol Buffers, MessagePack, INI files, SQL dumps, HAR files - critical for engineering workflows. The deeper your understanding of file formats, the faster you grow as a Data Engineer, Analyst, or ML Engineer. Tools change, formats stay.

𝗗𝗮𝘁𝗮 𝗔𝗻𝗮𝗹𝘆𝘀𝗶𝘀 𝗕𝗲𝗴𝗶𝗻𝘀 𝗕𝗲𝗳𝗼𝗿𝗲 𝗬𝗼𝘂 𝗘𝘃𝗲𝗻 𝗦𝘁𝗮𝗿𝘁 𝗔𝗻𝗮𝗹𝘆𝘇𝗶𝗻𝗴: 𝗧𝗵𝗲 𝗜𝗺𝗽𝗼𝗿𝘁𝗮𝗻𝗰𝗲 𝗼𝗳 𝗣𝗿𝗼𝗽𝗲𝗿 𝗗𝗮𝘁𝗮 𝗠𝗮𝗻𝗮𝗴𝗲𝗺𝗲𝗻𝘁 Data collection is just as crucial as the data analysis stage—if not more so. The quality of your data analysis and the insights you derive depend heavily on the quality of your data. I recently experienced this firsthand during a data analysis project. While the data was stored in a database, there was little to no documentation about the tables or a proper data dictionary. Field names were inconsistent across different tables, and there were no constraints or data validation rules to ensure uniformity in what could be stored in each column. As you can imagine, this made the job extremely tedious. I had to spend significant time figuring things out—mapping fields, identifying relationships, and cleaning up inconsistencies—before I could even begin analyzing the data. 𝗧𝗼 𝗮𝘃𝗼𝗶𝗱 𝘀𝗶𝘁𝘂𝗮𝘁𝗶𝗼𝗻𝘀 𝗹𝗶𝗸𝗲 𝘁𝗵𝗶𝘀, 𝗵𝗲𝗿𝗲 𝗮𝗿𝗲 𝘀𝗼𝗺𝗲 𝗯𝗲𝘀𝘁 𝗽𝗿𝗮𝗰𝘁𝗶𝗰𝗲𝘀 𝗳𝗼𝗿 𝗯𝗲𝘁𝘁𝗲𝗿 𝗱𝗮𝘁𝗮 𝗺𝗮𝗻𝗮𝗴𝗲𝗺𝗲𝗻𝘁: ⭐⭐ 𝗗𝗲𝘃𝗲𝗹𝗼𝗽 𝗮 𝗗𝗮𝘁𝗮 𝗗𝗶𝗰𝘁𝗶𝗼𝗻𝗮𝗿𝘆 – Ensure every database has a comprehensive data dictionary that describes each table, field, and the relationships between them. ⭐⭐ 𝗦𝘁𝗮𝗻𝗱𝗮𝗿𝗱𝗶𝘇𝗲 𝗙𝗶𝗲𝗹𝗱 𝗡𝗮𝗺𝗲𝘀 – Use consistent naming conventions across all tables to prevent confusion. ⭐⭐ 𝗜𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁 𝗖𝗼𝗻𝘀𝘁𝗿𝗮𝗶𝗻𝘁𝘀 𝗮𝗻𝗱 𝗩𝗮𝗹𝗶𝗱𝗮𝘁𝗶𝗼𝗻 – Set rules for what type of data can be stored in each field (e.g., data types, ranges, required fields). ⭐⭐ 𝗘𝗻𝘀𝘂𝗿𝗲 𝗖𝗼𝗹𝗹𝗮𝗯𝗼𝗿𝗮𝘁𝗶𝗼𝗻 𝗕𝗲𝘁𝘄𝗲𝗲𝗻 𝗧𝗲𝗮𝗺𝘀 – The data collection and database management teams should work closely with data analysts to ensure seamless transitions. ⭐⭐ 𝗜𝗻𝘃𝗲𝘀𝘁 𝗶𝗻 𝗣𝗿𝗼𝗽𝗲𝗿 𝗧𝗿𝗮𝗶𝗻𝗶𝗻𝗴 – Equip teams responsible for data collection and management with the skills to organize and structure data effectively. Remember, messy data leads to messy insights. When data is well-organized and properly documented, it not only saves time but also ensures the accuracy of your analysis. 𝗪𝗵𝗮𝘁 𝘀𝘁𝗲𝗽𝘀 𝗱𝗼 𝘆𝗼𝘂 𝘁𝗮𝗸𝗲 𝘁𝗼 𝗺𝗮𝗶𝗻𝘁𝗮𝗶𝗻 𝗱𝗮𝘁𝗮 𝗾𝘂𝗮𝗹𝗶𝘁𝘆 𝗶𝗻 𝘆𝗼𝘂𝗿 𝘄𝗼𝗿𝗸? 𝗟𝗲𝘁 𝗺𝗲 𝗸𝗻𝗼𝘄 𝗶𝗻 𝘁𝗵𝗲 𝗰𝗼𝗺𝗺𝗲𝗻𝘁𝘀! 

For early stage biotech startups, building data infrastructure often seems overwhelming. But there are a few simple things that can make a huge difference early on. The common thread between all of these is that they encourage consistency without requiring additional technical tools. That means you can get started immediately, but the consistency will create a foundation that makes it 10x easier to adopt the technical tools when you're ready for them. 1) Agree on naming conventions. Do this for experiment names, assay names, and anything else that different teams will be defining and then communicating about. Yes, getting everyone to agree and stay consistent is kind of a pain. But it will pay off many times over down the road, particularly when new people join your team. 2) Keep lists of all these things your teams are naming in a centrally accessible place. A Google doc or an Excel file in Sharepoint will do. Eventually these lists will get too long to manage this way, but you can cross that bridge when you get to it. It's much easier to switch to a database from a spreadsheet than from not even having a spreadsheet. 3) Pick one place to put your data, with a consistent organizational structure, and make sure everyone puts it there. It's fine if that's Sharepoint or Google Drive. Having all your data organized in Sharepoint is infinitely better than having it scattered across multiple systems, no matter how much better suited they might seem. Use the naming conventions you defined above in the directory names for consistency. 4) Write all these conventions in a shared document that's easy to find. Then regularly remind your team to review it. Yes, you're going to outgrow all of these processes. But until then they'll build an organized foundation for the technical tools that you'll eventually replace them with. Yes, these are all kind of a pain to decide on then do consistently. But do you know what's an even bigger pain? Cleaning up the mess after years of not doing them. And the best part is you can start doing all of them immediately. So don't wait. Start building a foundation today and thank yourself later.

Can you imagine driving around in a town where every street sign used a different font, size, shape, and color, regardless of type? Green and purple stop signs, brown and blue warning signs, all different. 😕 Chaos, right? That's exactly what happens in your HubSpot CRM when data isn't standardized! 😱 Inconsistent formatting is one of the biggest data hygiene challenges we face. Phone numbers, names, dates - they're like a digital Tower of Babel, each speaking its own language. It's confusing and it's a hidden profit killer. From an anthropological standpoint, this mirrors the diversity of human cultures. 🤔 Just as different societies developed unique ways of recording information, our teams often develop their own data entry "dialects." 🌍 But in our global digital village, we need a common language. That's where data standardization comes in! 📊 The Art of Data Standardization: Best Practices to Bring Order to Chaos 🎨 Establish clear guidelines for data entry. Create a "data culture" within your organization, complete with its own norms and practices. It's like creating a shared language for your team! Here are some simple, but powerful best practices: ✔️ Set standards for formats (dates, phone numbers, states, etc.) ✔️ Use consistent capitalization and abbreviation ✔️ Using pre-set drop-downs vs open text fields whenever possible ✔️ Set consistent data structure formats and avoid duplicate fields for the same information - eg. pick between "First Name", "Last" as separate fields vs "Customer Name" as a single field ✔️ Define required vs. optional fields Remember, standardization isn't about stifling creativity - it's about creating a shared understanding that empowers everyone to communicate effectively. 💪 In the end, standardized data is like a well-organized library. It makes finding and using information a breeze, leading to better insights and stronger customer relationships. 📚🚀 #DataStandardization #HubSpotCRM #DigitalCulture #DataHygiene --- 👋🏼 Hi, I'm Omi, co-founder of Diaz & Cooper, a Platinum HubSpot Solutions Partner helping B2B companies create efficient revenue operations. I'm on a mission to bring the human back to HubSpot.

Best Practices for Cross-Domain Data Discoverability and Standardization in Data Mesh To operationalize Data Mesh successfully, organizations must focus on practical, repeatable actions that balance domain autonomy with enterprise-wide collaboration. Below are actionable best practices, grounded in real-world implementations, for tackling discoverability and standardization.   1. Best Practices for Data Discoverability a. Centralized Metadata Catalog with Domain Ownership Action: ·     Build a global metadata catalog that aggregates metadata from all domains. ·     Mandate domains to own and enrich metadata (e.g., data lineage, business definitions, PII tags). Example: ·     Intuit uses DataHub to federate metadata from domains like TurboTax (tax data) and QuickBooks (accounting). Each domain team tags datasets with business context (e.g., “customer lifetime value”) and technical metadata (e.g., freshness, schema). ·     Outcome: Data consumers can search for terms like “customer revenue” and instantly find relevant datasets across domains. Tools: ·     DataHub, Amundsen, AWS Glue Data Catalog, Alation.   b. Domain-Specific Data Product Interfaces Action: ·     Require domains to expose data products via standardized interfaces. ·     Publish clear data product SLAs.   c. Semantic Search & Tagging Action: ·     Implement business-friendly tagging. ·     Use natural language search.   2. Best Practices for Data Standardization a. Federated Governance with Guardrails Action: ·     Define global standards (e.g., data formats, unique identifiers) but let domains choose tools. ·     Use automated policy-as-code to enforce standards (e.g., schema validation).   b. Self-Service Data Product Templates Action: ·     Provide pre-built templates for domains to create data products. ·     Include automated quality checks.   c. Domain Collaboration Contracts Action: ·     Establish data contracts between producing and consuming domains. ·     Use version control for schemas and APIs.   d. Automated Data Quality Monitoring Action: ·     Embed quality checks into domain pipelines (e.g., freshness, accuracy). ·     Publish quality metrics to the metadata catalog.   Lessons from the Trenches 1 Start Small, Scale Gradually 2 Automate Compliance 3 Culture > Tools     Details with examples and tools for each best practices and other information are available in our Daily Premium Content Newsletter.   Image Source: AWS   Transform Partner – Your Digital Transformation Consultancy

Making building data useful begins with establishing metadata standards, enforcing them across your portfolio, and then keeping that up to date. A practical framework for this comes from Mapped 's Chief Data Officer Jason Koh... Here's his five-step data modeling process that building owners can adopt (whether using Mapped’s tools or otherwise): • 𝗧𝘆𝗽𝗲 𝗔𝘀𝘀𝗶𝗴𝗻𝗺𝗲𝗻𝘁: Identify and tag each data point with what kind of thing it is (e.g. zone temperature sensor, discharge air damper command, etc.).    • 𝗜𝗱𝗲𝗻𝘁𝗶𝗳𝗶𝗰𝗮𝘁𝗶𝗼𝗻: Parse and interpret identifiers in the raw data to group related points and devices. From “VAV101_RM100_ZN_T”, you can infer and confirm that this is the zone temp for VAV unit 101 serving room 100.    • 𝗟𝗶𝗻𝗸𝗶𝗻𝗴: Establish relationships between entities—which points belong to which piece of equipment, which equipment serves which space, and how equipment is nested.    • 𝗨𝗻𝗶𝗳𝗶𝗰𝗮𝘁𝗶𝗼𝗻: Reconcile disparate data sources into a unified model. A single “real world” object might appear in the BAS, in mechanical schedules, and in drawing files, all with different names. Unification means mapping those together.    • 𝗘𝗻𝗿𝗶𝗰𝗵𝗺𝗲𝗻𝘁: Add any additional context or custom metadata needed for your use cases. This could be human-friendly equipment labels, associations to floor or zone names, capacities, control sequences, commissioning dates, etc.     By breaking the problem into these steps (or similar ones), owners (and their vendors) can systematically build a robust data model. That's what our latest 𝗡𝗲𝘅𝘂𝘀 𝗟𝗮𝗯𝘀 deep dive is all about... https://lnkd.in/gri63eqw I spoke to experts 𝗝𝗮𝘀𝗼𝗻 𝗞𝗼𝗵 of 𝗠𝗮𝗽𝗽𝗲𝗱, 𝗦𝘁𝗲𝗽𝗵𝗲𝗻 𝗗𝗮𝘄𝘀𝗼𝗻-𝗛𝗮𝗴𝗴𝗲𝗿𝘁𝘆 of 𝗡𝗼𝗿𝗺𝗮𝗹 𝗦𝗼𝗳𝘁𝘄𝗮𝗿𝗲, 𝗔𝗻𝗱𝗿𝗲𝘄 𝗥𝗼𝗱𝗴𝗲𝗿𝘀 of 𝗔𝗖𝗘 𝗜𝗼𝗧 𝗦𝗼𝗹𝘂𝘁𝗶𝗼𝗻𝘀, and 𝗕. 𝗦𝗰𝗼𝘁𝘁 𝗠𝘂𝗲𝗻𝗰𝗵 of 𝗝𝟮 𝗜𝗻𝗻𝗼𝘃𝗮𝘁𝗶𝗼𝗻𝘀 - 𝗮 𝗦𝗶𝗲𝗺𝗲𝗻𝘀 𝗖𝗼𝗺𝗽𝗮𝗻𝘆 to get their take. But what do you think?

Which Data File Format to Use? CSV, JSON, Parquet, Avro, ORC Choosing the right data file format can significantly impact the efficiency and performance of your data workflows. Here’s a quick guide to help you decide: 1. **CSV (Comma-Separated Values):** - **Pros:** Simple, human-readable, and easy to use. - **Cons:** Lack of schema, larger file sizes, no support for complex data types. - **Best For:** Simple datasets, quick data interchange. 2. **JSON (JavaScript Object Notation):** - **Pros:** Human-readable, supports nested data structures, widely used in web applications. - **Cons:** Larger file sizes compared to binary formats, parsing can be slow. - **Best For:** Web APIs, documents with nested structures. 3. **Parquet:** - **Pros:** Columnar storage, efficient for read-heavy operations, supports complex data types. - **Cons:** Not human-readable, requires more processing power for writing. - **Best For:** Big data processing, analytical queries. 4. **Avro:** - **Pros:** Compact, efficient serialization, supports schema evolution, binary format. - **Cons:** Not human-readable, requires schema for reading. - **Best For:** Data interchange between systems, streaming data. 5. **ORC (Optimized Row Columnar):** - **Pros:** Columnar storage, highly efficient for large datasets, strong compression. - **Cons:** Not human-readable, best suited for Hadoop ecosystems. - **Best For:** Big data storage, batch processing. In summary, your choice depends on your specific use case: - For simplicity and ease of use: **CSV or JSON** - For performance and efficiency in big data: **Parquet or ORC** - For data interchange and schema evolution: **Avro** What’s your preferred data format and why? Share your thoughts! #DataEngineering #BigData #DataScience #DataFormats #CSV #JSON #Parquet #Avro #ORC

𝗠𝗮𝘀𝘁𝗲𝗿𝗶𝗻𝗴 𝘁𝗵𝗲 𝗥𝗶𝗴𝗵𝘁 𝗙𝗶𝗹𝗲 𝗙𝗼𝗿𝗺𝗮𝘁𝘀 𝗶𝗻 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 Choosing the correct file format can make or break your data engineering project! Here’s a field guide, with real job scenarios, to help you pick wisely: 1. 𝘾𝙎𝙑: Think of CSVs as your everyday bank statement - clean, simple rows and columns. Great for quick logs and spreadsheets. 2. 𝙅𝙎𝙊𝙉: Ever pulled data from an API like Twitter or LinkedIn? Chances are, you’ve encountered JSON - a favorite for nested, complex data that’s readable for both people and machines. 3. 𝙋𝙖𝙧𝙦𝙪𝙚𝙩: Retail giants (like Walmart) store massive sales data in Parquet. Why? It’s lightning-fast for analytics and saves money on cloud storage. 4. 𝘼𝙫𝙧𝙤: Streaming platforms (think Netflix) use Avro to handle massive real-time data and logs. Its flexibility supports schema evolution, which keeps things smooth as data needs change. 5. 𝙊𝙍𝘾: Big data warehouses (for example, those running Hive) pick ORC files for efficient storage and quick, large-scale analytics—especially at enterprise scale. 5. 𝙓𝙈𝙇: Old but gold! Legacy healthcare and insurance systems still rely on XML for structured, secure data exchanges and system integration. 𝐏𝐫𝐨 𝐭𝐢𝐩: The right file format = faster data, simpler pipelines, lower costs, and better insights! Which formats do you rely on in your daily projects? 👇 Share your experience or bookmark this for later. #DataEngineering #BigData #CSV #JSON #Parquet #Avro #ORC #XML #CareerGrowth #TechTips

When systems speak the same language, clinical trials move faster. Here’s a quick tip to optimize clinical trials — be consistent in your naming standards across study platforms like EDC, IRT, RTSM, central labs, and beyond. “Randomized_At”, “Randomization_Date”, “Date_of_Randomization”, “Randomized_Date” If each system in your study refers to the same data point with a different name, you’re adding unnecessary complexity for your Clinical Operations and Data Management teams — especially when reconciling data across systems. And it doesn’t stop there. For visit labels, one system might use “Visit”, another “Visit Name”, and another “Visit_Name.” Even the values may vary: “Visit 1 Day 1”, “Visit 2 Day 7”, Visit 3 Day 14” vs. “Day 1 Visit 1”, “Day 7 Visit 2”, “Day 14 Visit 3”. These inconsistencies then propagate into every dependent field, turning what should be straightforward mapping into a manual, error-prone process across your entire study. If you want transparency and visibility throughout your study, data consistency is key. Naming conventions are often an afterthought in the race to start a trial. But they are foundational to faster execution, data integrity, and reducing the burden on the study teams doing the work. Let’s set them up for success from the very start.