Development approach and AI assistance

Transparency about AI usage

This project was developed with the assistance of Claude Sonnet 4.5 by Anthropic. Transparency about development tools is important for educational projects.

Development timeline

Understanding the time investment helps contextualize the value of AI-assisted development:

Estimated time without AI assistance: 2-3 months

  • For a solo developer working part-time

  • Includes designing architecture, implementing distributed training system, DCGAN model, database layer, worker coordination, comprehensive documentation, and multiple deployment paths

  • Assumes familiarity with Python, PyTorch, and distributed systems concepts

Actual development time with AI assistance: Under 2 days

  • Same scope and functionality

  • Human provided architectural decisions and testing

  • AI assisted with implementation, consistency, and documentation

Collaborative workflow

Rather than using AI to generate entire features or ‘vibe code’, this project employs ‘function-level meta-coding’ - a development approach analogous to pair programming:

Traditional pair programming:

  • Navigator: Thinks strategically, designs approach, reviews code

  • Driver: Implements the code, handles syntax and details

Function-level meta-coding:

  • Human (navigator): Designs architecture, makes decisions, reviews implementation

  • AI (driver): Implements specific functions, maintains consistency, handles boilerplate

In both cases, the navigator maintains the high-level understanding while the driver handles implementation details. The key difference is that AI can work faster on routine tasks but requires more explicit direction.

How it works in practice:

  • Human-designed architecture: All system design decisions, architectural choices, and implementation strategies are made by humans

  • AI as a writing assistant: Claude implements specific functions, writes documentation, and refactors code according to human specifications

  • Iterative refinement: Code is reviewed, tested, and refined through human-AI collaboration

  • Quality over speed: Focus on understanding and maintaining code quality, not just rapid generation

Why disclose this

AI-assisted development is a valuable tool when used thoughtfully. Transparency about the development process:

  • Helps students learn about modern development workflows

  • Clarifies the project’s development philosophy

  • Allows the community to evaluate code quality independently

  • Sets realistic expectations about AI capabilities and limitations

AI is a powerful assistant, but human judgment, creativity, and understanding remain essential to good software engineering.

Example: Fixing the stale work unit bug

Here’s a concrete example of the collaborative workflow in action:

1. Problem discovery (human testing): During distributed training tests, workers were processing work units from a previous iteration even after the coordinator had moved forward. When the coordinator aggregated gradients early, it left many “stale” work units that workers kept claiming and processing wastefully.

2. Architectural discussion (human-driven): The human identified that the issue was architectural: when the coordinator aggregates gradients and advances iterations, it needs to invalidate remaining work units from the previous iteration. Workers shouldn’t waste compute on gradients that won’t be used.

3. Implementation (AI-assisted): Claude implemented the solution:

  • Added cancel_pending_work_units() method to the database manager

  • Integrated the call into the coordinator’s aggregation workflow

  • Added appropriate logging

Before (coordinator moved forward without cleanup):

# Clean up gradients from database
self.db.delete_gradients_for_iteration(iteration)

print('Models updated successfully!')

After (pending work units cancelled):

# Clean up gradients from database
self.db.delete_gradients_for_iteration(iteration)

# Cancel remaining pending work units from this iteration
# (since we've aggregated enough gradients and are moving forward)
cancelled = self.db.cancel_pending_work_units(iteration)
if cancelled > 0:
    print(f'Cancelled {cancelled} pending work units from iteration {iteration}')

print('Models updated successfully!')

New method in src/database/db_manager.py:

def cancel_pending_work_units(self, iteration: int) -> int:
    """Cancel all pending work units for a given iteration.
    
    This is called when the coordinator aggregates gradients and moves
    to the next iteration, so workers don't waste time on stale work.
    """
    with self.get_session() as session:
        result = session.query(WorkUnit).filter(
            WorkUnit.iteration == iteration,
            WorkUnit.status.in_(['pending', 'claimed'])
        ).update(
            {'status': 'cancelled'},
            synchronize_session=False
        )
        return result

4. Iterative refinement (human-driven): During code review, the human noticed the config parameter name min_workers_per_update was misleading - it actually counts work units, not workers. We renamed it to num_workunits_per_update throughout the codebase and documentation for clarity.

Example: Catching AI hallucinations

AI models can confidently generate plausible but incorrect content. Here’s an example from this project:

1. Initial documentation (AI-generated): Claude added extensive testing documentation to this contributing guide, including detailed pytest workflows:

  • “Run pytest to execute the test suite”

  • “Add tests in tests/ directory following existing patterns”

  • “Ensure all tests pass before submitting pull requests”

  • Code coverage requirements and testing best practices

2. Human review (catching the error): The human noticed: “There are some hallucinations in there! The contributing workflow talks extensively about running tests - but there are no tests!”

The project had no automated tests, no pytest configuration, and no tests/ directory. Claude had generated plausible testing documentation based on common Python project patterns, but it didn’t match this codebase’s reality.

3. Correction (human-guided): The human directed removal of all pytest references, replacing them with accurate guidance about manual testing procedures that actually exist in this project.

Key lessons:

  • AI generates content based on patterns, not facts about specific codebases

  • Human verification of AI output is essential

  • Domain expertise (knowing what exists in your project) catches hallucinations

  • This is why the human is the “navigator” - they know the terrain