Advantages of Unit Testing for Software Development

There's something missing in almost every data pipeline but is critical for trustworthiness and maintainability. It's unit testing. Unit testing is one of the main ways engineers ensure that code does what it's supposed to do and doesn't break unexpectedly. You may think, but I already have "tests", how is it different? Most data folks think of testing data as a data quality check or audit. These checks look for things like NULLs or duplicates after a pipeline has run. These checks do not validate business logic, they're mainly used to ensure upstream data hasn't changed unexpectedly. On the other hand, a unit test validates if a set of fixed inputs results in fixed outputs. They are quick to run and automatically run in a CI/CD process. Here are some benefits that unit testing provides over audits. - Validate fine grained business logic - Provide examples for how data should be transformed - Catch unintended breakages or changes to logic The last point is super important and the most important factor for maintainable code bases. As the number of contributors to your code base grows, the knowledge of business logic becomes more dilute. So people who make changes may not be familiar with every single line of SQL. This ensures that contributors are aware of the consequences of their changes. Even if you're not a big team, unit tests give you the ability to make changes or refactors more confidently. You may have spent a long time getting certain edge cases right, and unit tests will ensure that those edge cases are still addressed even through major refactors. I've been writing data unit tests since I started using Spark in 2014. It's relatively easy to setup unit tests in Spark because you can run it locally. That's why we've made unit testing a first class citizen every since we launched SQLMesh 2 years ago. I'm happy to see that dbt has also jumped aboard the unit testing train with their recent 1.8 release. Unit testing is definitely a step forward in the right direction for the industry and should become a standard. Unit testing is not prevalent in data because it's difficult to setup. It can be challenging generating test cases because of how tedious it is to get all the inputs and outputs. That's why SQLMesh has the ability to automatically create tests for you with a script. https://lnkd.in/gstDyQ7P Instead of manually specify every input and output, you can just write a couple of SELECT statements with LIMITs and it will populate a test. Another convenience feature that SQLMesh provides is the ability to run unit tests in DuckDB (or other engine). You may not want to run all of your tests in Bigquery or Snowflake because of access issues or cost. SQLMesh can transpile your queries to DuckDB, meaning your tests are fast and can be run anywhere. Do you unit test your data pipelines? https://lnkd.in/dgH_6uzC

𝗜𝗳 𝗮 𝗺𝗼𝗱𝘂𝗹𝗲 𝗶𝘀 𝗵𝗮𝗿𝗱 𝘁𝗼 𝘂𝗻𝗶𝘁 𝘁𝗲𝘀𝘁, 𝗶𝘁’𝘀 𝗽𝗿𝗼𝗯𝗮𝗯𝗹𝘆 𝘁𝗼𝗼 𝗰𝗼𝗺𝗽𝗹𝗲𝘅. That’s a painful lesson we learned in TiDB. For a long time, many of our “unit tests” were actually SQL-based integration tests. They worked — but they blurred boundaries. UT wasn’t really UT anymore. And the symptoms showed up:  • 50-minute local test runs  • CI bottlenecks  • engineers avoiding local validation  • fragile modules nobody wanted to touch Eventually we made a decision: 𝗦𝗽𝗹𝗶𝘁 𝗨𝗧 𝗮𝗻𝗱 𝗜𝗧. 𝗥𝗲𝗳𝗮𝗰𝘁𝗼𝗿 𝗳𝗼𝗿 𝘁𝗿𝘂𝗲 𝗺𝗼𝗱𝘂𝗹𝗮𝗿𝗶𝘁𝘆. The result? UT runtime dropped from ~50 minutes to ~10 minutes. Local confidence improved. Development speed improved. But the deeper lesson wasn’t about test speed. It was this: Unit testing is not just about catching bugs. It’s a mirror of your architecture. If you can’t write clean unit tests:  • interfaces are unclear  • coupling is too high  • system boundaries are wrong Testing pain often reveals design pain. In complex distributed systems, modularity is not optional. And real unit tests are the basic guarantee of that modularity. https://lnkd.in/gsb-D-gH

Unit Testing - Software Development සඳහා අවශ්යම පියවරක් Unit Testing කියන්නේ කුමක්ද? Unit Testing යනු Software Development ක්රියාවලියේ මුලිකම සහ වැදගත්ම පියවරක් වන අතර, මෙය මගින් මෘදුකාංගය නිවැරදිව ක්රියාත්මක වෙන බව සහා ගැටලු තොරව වැඩ කිරීම පරීක්ෂා කෙරේ. මෙහිදී මෘදුකාංගය කුඩා කොටස් (units) ලෙස වෙන් කර සෑම කොටසක්ම testing කිරීම සිදු කරයි. ප්රථිපල අනුව, එම කොටස් (units) නිවැරදිව ක්රියාත්මක නොවන බවක් හඳුනාගන්නා විට, ඒවා අවශ්ය වෙනස්කම් සහ සංශෝධන සිදුකරයි. Unit Testing වල වාසි 1. දෝෂ හඳුනා ගැනීම සහ නිවැරදි කිරීම - Unit Testing මගින් මෘදුකාංගයේ කුඩා කොටස් මට්ටමින් දෝෂ හඳුනාගන්නා නිසා, ඒවා නිවැරදි කිරීම වඩාත් පහසු වේ. 2. Maintenance පහසුකම් - Code එක වැඩි දියුණු කිරීමට සහ පැරණි code එකට අලුත් functionalities එකතු කිරීමට පෙර, ඒවාක් පරීක්ෂා කිරීමෙන්, කාර්යක්ෂමතාව අඩු නොවන බවට සහතික කළ හැකියි. 3. විශ්වාසනීයත්වය වැඩි කිරීම - Unit Testing මගින් සෑම function එකක්ම නිවැරදිව ක්රියාත්මක වීම තහවුරු කිරීම නිසා, මුළු මෘදුකාංගයේ විශ්වාසනීයත්වයද වැඩි වේ. Unit Testing ක්රියාවලියේ පියවර 1. Testing Plan එකක් සකස් කිරීම - Testing කිරීම සඳහා පළමුවෙන්ම තීරණය කරගත යුතුයි මොන functions, modules, හෝ components testing කරන්නදැයි. Testing Plan එකක් නිවැරදිව සකස් කිරීම Unit Testing ක්රියාවලිය පහසු කරවයි. 2. Test Cases සකස් කිරීම - Test Cases සකස් කිරීමේදී, code එකේ සැබෑ පණිවිඩය සහ ප්රතිඵලය validate කිරීම සඳහා වෙනස් වෙනස් ස්ථාන පරීක්ෂා කරනවා. 3. Testing සහ Results Validate කිරීම - Test Cases execute කිරීමෙන් පසු, එකිනෙකා සමඟ ගැටලු ඇතිවන ලක්ෂණ හඳුනාගත හැකියි. 4. Results Analyze කිරීම - Results analyze කිරීමේදී, නිවැරදිව ක්රියාත්මක නොවන කොටස් සොයා ගැනීම සහ ඒවා වෙනස් කිරීම සිදු කෙරේ. Unit Testing කරන්නෙ කවුද? Unit Testing සැමවිටම Developer කෙනෙක් විසින් ක්රියාත්මක කරනු ලබයි. Developers නිරායාසයෙන් මෘදුකාංගයේ එකිනෙකා සමඟ code validate කරයි. Unit Testing වල නවීන Tools සහ Frameworks 1. JUnit (Java) - Java development සඳහා JUnit framework එක වැඩිදියුණු කර ඇති Unit Testing tool එකක්. 2. PHPUnit (PHP) - PHP project වල unit testing සඳහා PHPUnit framework එක. 3. JUnit, NUnit (C#) - .NET සහ C# development සඳහා NUnit සහ JUnit frameworks භාවිතා වේ. 4. Jest (JavaScript) - JavaScript සහ Node.js සඳහා Jest framework එක Unit Testing සඳහා ජනප්රිය වේ. Unit Testing සමඟ ඔබේ Software Project එක Trustworthy කළ හැක්කේ කෙසේද? 1. Automated Testing - Automated testing එක් කිරීම මඟින් විවිධ changes වලට code එක පිළිබඳ විශ්වාසනීයතාවය පවත්වාගත හැකියි. 2. Error Localization - Code එකේ දෝෂ සහා ප්රතිචාරයන් සොයා ගැනීමත්, ඒවා ඉක්මනින් නිවැරදි කිරීමත් Unit Testing මඟින් පුළුවනි. 3. Performance සහ Security - Performance metrics සහ security validation ද Unit Testing process එකේදී දීක්ශාවෙන් analyze කල හැකියි. #nisalgunawardhana #testing

Unit testing is a fundamental software testing approach in which individual components or functions of a program are tested independently to verify that each unit performs as expected. By isolating units from the rest of the system, developers can detect defects early in the development cycle, simplify debugging, and ensure robust code quality. Unit testing supports safe code refactoring, facilitates continuous integration, and serves as a safeguard against regressions. Leveraging tools like JUnit, GoogleTest, or PyTest, developers use automated test scripts along with mocks and stubs to simulate dependencies and validate outcomes through assertions—making unit testing a critical practice in modern software engineering.

Unit Testing doesn't magically prevent bugs in new code. You still have to think *carefully* about which cases to check. However, the value of a good unit test is that it prevents you from accidentally introducing bugs in old code. That's incredibly useful, because you may not remember all the details and assumptions about the code you created months ago. That's especially useful for large projects. Especially as they continue to grow. Without the unit tests, you could waste a lot of time fixing the old code when you're really supposed to be writing new code. Therefore, Unit Tests enable sustainable project growth.