Skip to content

Action Plan: Fixing Editable Imports in Databricks

For the User Experiencing the Import Issue

Immediate Actions (Do This Now)

  1. Apply the patches:
# In your Databricks notebook:
%pip install -q dbx-patch

from dbx_patch import patch_dbx
patch_dbx(verbose=True)
  1. Try your import again:
from testx import function1
print(function1())
  1. If it works → Make it permanent:
from dbx_patch import patch_and_install
patch_and_install()  # Installs sitecustomize.py, restarts kernel
  1. If it doesn't work → Run diagnostics:
# Enable debug mode
import os
os.environ['DBX_PATCH_DEBUG'] = '1'

# Try import again (watch for debug output)
from testx import function1

Upload and run: notebooks/diagnostic_editable_imports.ipynb

For the Developer (Repository Maintainer)

Completed Work ✅

  1. Deep Analysis:
  2. Identified all 5 layers of Databricks import hooks
  3. Documented how each layer blocks editable imports

  4. Created technical analysis: docs/docs/files/editable-install-analysis.md

  5. Missing Patches Added:

  6. wsfs_path_finder_patch.py - Verifies WsfsPathFinder compatibility
  7. post_import_hook_verify.py - Verifies PostImportHook compatibility

  8. Updated patch_dbx() to include verification steps

  9. Setup Automation:

  10. scripts/setup_dbx_patch.py - Full-featured Python setup script
  11. scripts/setup_dbx_patch.sh - Simple bash setup script
  12. notebooks/setup_quick_start.ipynb - Interactive setup notebook

  13. scripts/README.md - Complete documentation for all scripts

  14. Diagnostic Tools:

  15. notebooks/diagnostic_editable_imports.ipynb - Step-by-step diagnostics
  16. Debug mode support via DBX_PATCH_DEBUG environment variable

  17. Comprehensive verification functions

  18. Documentation:

  19. docs/docs/files/solution-guide.md - Complete troubleshooting guide
  20. docs/docs/files/editable-install-analysis.md - Technical deep dive
  21. scripts/README.md - Script usage guide
  1. Test with actual user's environment:
  2. Get access to their Databricks workspace
  3. Run diagnostic_editable_imports.ipynb
  4. Identify which specific hook is blocking

  5. Create targeted fix if needed

  6. Add unit tests:

# tests/unit/test_all_patches.py
def test_wsfs_path_finder_patch():
    from dbx_patch.patches.wsfs_path_finder_patch import patch_wsfs_path_finder
    result = patch_wsfs_path_finder(verbose=False)
    assert result.success or not result.hook_found  # OK if not in Databricks
  1. Add integration tests:
# tests/integration/test_editable_import.py
def test_editable_package_import():
    # Install test package as editable
    # Apply patches
    # Verify import works
  1. Create example package:
  2. Add examples/test_package/ with a simple Python package
  3. Include in CI/CD to test editable installs

  4. Document as reference implementation

  5. Update main README.md:

  6. Add quickstart section
  7. Link to setup scripts

  8. Add troubleshooting section

  9. Version and release:

  10. Tag current version
  11. Update CHANGELOG
  12. Publish to PyPI if not already done
  13. Create GitHub release with notes

Implementation Checklist

Patches (All Implemented ✅)

  • sys_path_init_patch.py - Process .pth files
  • wsfs_import_hook_patch.py - Allow editable paths
  • python_path_hook_patch.py - Preserve paths
  • autoreload_hook_patch.py - Register allowlist
  • wsfs_path_finder_patch.py - Verify compatibility
  • post_import_hook_verify.py - Verify compatibility

Scripts (All Created ✅)

  • scripts/setup_dbx_patch.py - Python setup script
  • scripts/setup_dbx_patch.sh - Bash setup script
  • scripts/README.md - Script documentation

Notebooks (All Created ✅)

  • notebooks/diagnostic_editable_imports.ipynb - Diagnostics
  • notebooks/setup_quick_start.ipynb - Quick setup

Documentation (All Created ✅)

  • docs/docs/files/editable-install-analysis.md - Technical analysis
  • docs/docs/files/solution-guide.md - Troubleshooting guide
  • scripts/README.md - Script usage

Testing (To Do)

  • Unit tests for each patch
  • Integration test for end-to-end workflow
  • CI/CD pipeline for automated testing
  • Example package for testing

Distribution (To Do)

  • Update main README.md
  • Update CHANGELOG
  • Create GitHub release
  • Verify PyPI package is up to date

Priority Order

P0 (Critical - Help User Now)

  1. ✅ Analyze root cause
  2. ✅ Verify existing patches cover all layers
  3. ✅ Create diagnostic notebook
  4. ✅ Create setup scripts
  5. → User runs patches and reports back

P1 (Important - Complete Solution)

  1. Get user feedback from running patches
  2. Fix any remaining issues
  3. Add tests
  4. Update README

P2 (Nice to Have - Polish)

  1. Create example package
  2. Improve CI/CD
  3. Add more examples
  4. Video tutorial

Files Created/Modified

New Files

  1. src/dbx_patch/patches/wsfs_path_finder_patch.py
  2. src/dbx_patch/patches/post_import_hook_verify.py
  3. scripts/setup_dbx_patch.py
  4. scripts/setup_dbx_patch.sh
  5. scripts/README.md
  6. notebooks/diagnostic_editable_imports.ipynb
  7. notebooks/setup_quick_start.ipynb
  8. docs/docs/files/editable-install-analysis.md
  9. docs/docs/files/solution-guide.md

Modified Files

  1. src/dbx_patch/patch_dbx.py - Added verification steps

Quick Reference: What Each Hook Does

Hook Location Purpose Blocks Editable? Patch Status
sys_path_init Initialization Modify sys.path Yes (doesn't process .pth) ✅ Patched
WsfsImportHook wsfs_import_hook.py Block non-workspace imports Yes ✅ Patched
WsfsPathFinder WsfsPathFinder.py Prevent notebook imports No ✅ Verified
PythonPathHook pythonPathHook.py Manage sys.path changes Yes (removes paths) ✅ Patched
AutoreloadHook autoreload/discoverability/ Wrap imports Yes ✅ Patched
PostImportHook PostImportHook.py Trigger callbacks No ✅ Verified

Expected Outcome

After applying patches, the following should work:

# Install package with uv
!cd /path/to/package && uv sync --active

# Apply patches
from dbx_patch import patch_dbx
patch_dbx()

# Import should now work!
from testx import function1
print(function1())  # ✅ SUCCESS

Contact Points

  • User needs help: Share scripts/setup_dbx_patch.py or notebooks/setup_quick_start.ipynb
  • Still not working: Request output from diagnostic_editable_imports.ipynb
  • Found a bug: File GitHub issue with diagnostics output
  • Need enhancement: Submit feature request with use case

Summary

All necessary code is complete. The user should:

  1. Run patch_dbx()
  2. Try their import
  3. If it fails, run diagnostics
  4. Report back with results

The most likely outcome is that it will work because all 5 hook layers are now patched/verified. If not, the diagnostic notebook will identify the specific blocking point.