{"name":"napari-chat-assistant","display_name":"Local Chat Assistant","visibility":"public","icon":"","categories":["Utilities"],"schema_version":"0.2.1","on_activate":null,"on_deactivate":null,"contributions":{"commands":[{"id":"napari-chat-assistant.chat_panel","title":"Local Chat Assistant","python_name":"napari_chat_assistant.widgets.chat_widget:chat_widget","short_title":null,"category":null,"icon":null,"enablement":null}],"readers":null,"writers":null,"widgets":[{"command":"napari-chat-assistant.chat_panel","display_name":"Chat Assistant","autogenerate":false}],"sample_data":null,"themes":null,"menus":{},"submenus":null,"keybindings":null,"configuration":[]},"package_metadata":{"metadata_version":"2.4","name":"napari-chat-assistant","version":"2.3.1","dynamic":["license-file"],"platform":null,"supported_platform":null,"summary":"Local AI and deterministic workbench for napari image-analysis workflows","description":"# napari-chat-assistant\n\n[![License MIT](https://img.shields.io/pypi/l/napari-chat-assistant.svg?color=green)](https://github.com/wulinteousa2-hash/napari-chat-assistant/raw/main/LICENSE)\n[![PyPI](https://img.shields.io/pypi/v/napari-chat-assistant.svg?color=green)](https://pypi.org/project/napari-chat-assistant)\n[![Python Version](https://img.shields.io/pypi/pyversions/napari-chat-assistant.svg?color=green)](https://python.org)\n[![napari hub](https://img.shields.io/endpoint?url=https://api.napari-hub.org/shields/napari-chat-assistant)](https://napari-hub.org/plugins/napari-chat-assistant)\n\nLocal, Ollama-powered AI and deterministic workbench for napari image-analysis workflows.\n\n`napari-chat-assistant` adds a dock widget inside napari that understands the active viewer session, runs built-in image-analysis actions, generates executable napari Python code when a request goes beyond the current toolset, and lets users promote repeatable tasks into one-click shortcuts.\n\nThe goal is not to bolt a generic chatbot onto a viewer. The goal is to turn napari into a more practical analysis workspace for people who work with microscopy and other large multidimensional imaging datasets, especially users who want local AI help, reproducible workflows, direct control over their data, and fewer clicks per task.\n\n## What's New In 2.3.1\n\nVersion `2.3.1` improves local-model performance diagnostics and reduces the live viewer context sent to Ollama.\n\nQuick feature summary:\n- Ollama prompt/generation metadata is now captured in local telemetry\n- telemetry summaries now include tokenization and local-model throughput diagnostics\n- the model request uses a compact viewer-context payload instead of verbose per-layer profiler evidence\n- Telemetry Log views show newest records first\n- `Copy Report` copies a shareable Markdown telemetry report to the clipboard\n\n## What's New In 2.3.0\n\nVersion `2.3.0` adds an optional local `Voice Input` workflow for users who prefer speaking prompts instead of typing them.\n\nThis update was added in response to feedback from Ron DeSpain on image.sc:\n- https://forum.image.sc/t/napari-chat-assistant-experimenting-with-a-local-chat-assistant-inside-napari-looking-for-feedback/120351/2\n\nQuick feature summary:\n- optional `Voice Input` entry under `Advanced`\n- local microphone recording with input-device selection\n- live green input-level meter during recording\n- local transcription with [`faster-whisper`](https://github.com/SYSTRAN/faster-whisper)\n- editable transcript preview before execution\n- direct `Run` from the voice window without an extra prompt-send step\n- reusable non-blocking voice window that stays open while you work in napari\n\nImportant:\n- voice input is optional and is not required for the base plugin\n- it requires [`faster-whisper`](https://github.com/SYSTRAN/faster-whisper) installed in the same Python environment as napari\n- microphone support also depends on the Qt multimedia stack available in that same environment\n\nFor complete release history, see [CHANGELOG.md](CHANGELOG.md).\n\n## Who It Is For\n\nThis plugin is built for:\n- imaging core facility users\n- researchers, staff scientists, and students working with imaging data\n- teachers and educators running imaging demos or training sessions\n- It is also designed for users who prefer to describe analysis goals in natural language instead of memorizing commands or writing scripts.\n\nIt is especially useful when you:\n- inspect large 2D or 3D image data in napari\n- move between interactive viewing, measurement, and Python-based analysis\n- want a local open-weight model instead of a cloud service\n- need to save, reuse, restore, and teach common imaging workflows\n- want fast deterministic actions and one-click shortcuts for repeated tasks\n\n## Why It Is Different\n\nThis plugin is built from practical imaging workflow needs, not from the idea of putting a generic chatbot beside a viewer.\n\nIt is designed around how work actually happens in napari:\n- start from the data already open in the viewer\n- inspect layers, objects, and regions of interest\n- run the next analysis step through chat, actions, templates, shortcuts, or code\n- review the result directly in the same session\n- save useful workflows for later reuse\n\nThe assistant is grounded in the live napari session. It can inspect loaded layers, use ROI context, run built-in analysis actions, generate or refine viewer-bound Python when needed, and support deterministic one-click workflows through `Actions` and `Shortcuts`. The result is closer to an imaging workbench than a chat panel.\n\n## Interaction Model\n\nThe plugin now supports a deliberate spectrum of interaction styles:\n- `Prompt`: AI-first natural language requests\n- `Code`: direct viewer-bound Python for users who want exact control\n- `Templates`: reusable examples and built-in starting points\n- `Actions`: deterministic built-in functions that can be run directly\n- `Shortcuts`: user-defined one-click action buttons for repeated daily work\n\nThis is now a core design principle of the plugin: reduce how many clicks and how much time it takes for a user to complete a task.\n\n## Interface Overview\n\n![Interface overview](docs/interface-overview.png)\n\nThe main dock is organized into a small number of practical work areas:\n\n1. `Connection and model controls`\n   Select the local model, monitor status, and manage the backend connection.\n\n2. `Layer Context`\n   Review the active workspace layers and insert exact layer names into prompts or code.\n\n3. `Library`\n   Browse reusable `Prompts`, `Code`, `Templates`, and deterministic `Actions`.\n\n4. `Shortcuts`\n   Keep your most-used actions as one-click buttons for repeated work.\n\n5. `Session`\n   Save or restore workspace state and access activity, telemetry, and diagnostics.\n\n6. `Chat`\n   See the assistant transcript, generated code, and direct action feedback in one place.\n\n7. `Prompt`\n   Enter natural language requests or paste Python for `Run My Code` and `Refine My Code`.\n\n## What You Can Do\n\nCurrent workflows include:\n- inspect the selected layer or named layers with structured summaries\n- review live layer context and insert exact layer names into prompts or code\n- run built-in tools for enhancement, thresholding, binary mask cleanup, measurement, projection, cropping, montage, presentation, and layer visibility control\n- add non-destructive annotation overlays, including free text, particle labels, callout labels with leader lines, and boxed titles above the image\n- use deterministic `Actions` for common workflows without depending on prompt phrasing\n- build and save your own `Shortcuts` layouts for repeated one-click work\n- inspect ROI context and measure or extract values from `Labels`, `Shapes`, and line-based workflows\n- use interactive analysis widgets such as `ROI Intensity Analysis`, `Line Profile Analysis`, and `Group Comparison Statistics`\n- access SAM2 setup, live preview, box prompting, points prompting, and mask refinement from the same workbench\n- browse built-in prompt templates, code templates, and learning templates for plugin workflows, teaching, and model testing\n- generate napari Python code when no built-in tool is the right fit\n- paste and run your own viewer-bound Python from the prompt box with `Run My Code`\n- repair or explain broken pasted Python with `Refine My Code`\n- save, pin, tag, rename, and reuse prompts and code from the local Library\n- use built-in synthetic data generators for repeatable testing, teaching, and workflow development\n- learn from built-in content for microscopy, electron microscopy, imaging physics, quantitative imaging, statistics, academic prompting, and language support\n- save and restore workspace state with a JSON manifest plus OME-Zarr assets for generated image and labels data\n- use `Atlas Stitch` from the advanced menu for specialized stitching and export workflows\n- use optional local `Voice Input` from `Advanced` for microphone recording, transcript review, and direct prompt execution\n\nExample requests:\n- `Inspect the selected layer`\n- `Preview threshold on the selected image`\n- `Apply gaussian blur to the selected image with sigma 1.2`\n- `Remove small objects from the selected mask with min_size 64`\n- `Run watershed on the selected mask`\n- `Measure labels table for the selected labels layer`\n- `Annotate template_blob_labels with particle 1 to 4`\n- `Annotate template_blob_labels using publication-style callout labels with leader lines.`\n- `Add title WT Group N=10 above the image on the left`\n- `Inspect the current ROI`\n- `Extract ROI values from the selected image using the current ROI`\n- `Open ROI intensity analysis`\n- `Initialize a SAM2 points prompt layer for the selected image`\n- `Write napari code to plot object area by condition`\n\n## Local-First By Design\n\nThe assistant runs on local open-weight models through Ollama:\n- no API key required\n- no cloud dependency\n- no internet requirement during normal use\n- no image data leaves your workstation\n\nThis makes it a better fit for research and facility environments where users want privacy, controllability, and local reproducibility.\n\n## Installation\n\nRequirements:\n- Python 3.9+\n- napari\n- Ollama running locally\n- one local Ollama model, such as `nemotron-cascade-2:30b`\n\nInstall Ollama and start the server:\n\nmacOS and Linux:\n\n```bash\ncurl -fsSL https://ollama.com/install.sh | sh\nollama serve\n```\n\nWindows:\n- download and install Ollama from `https://ollama.com/download/windows`\n- start the Ollama application or service\n\n```bash\nollama pull nemotron-cascade-2:30b\n```\n\nOptional model alternatives:\n\n```bash\nollama pull gpt-oss:120b\nollama pull qwen3-coder-next:latest\nollama pull qwen3.5\nollama pull qwen2.5:7b\n```\n\nInstall the plugin:\n\n```bash\npip install napari-chat-assistant\n```\n\nOptional voice input support:\n\n```bash\npip install faster-whisper\n```\n\nInstall `faster-whisper` in the same Python environment that launches napari if you want to use `Advanced -> Voice Input`.\n\nDevelopment install:\n\n```bash\ngit clone https://github.com/wulinteousa2-hash/napari-chat-assistant.git\ncd napari-chat-assistant\npip install -e .\n```\n\nThe plugin does not bundle Ollama or model weights. Larger models may require substantial RAM or VRAM.\n\n## Usage\n\n1. Start napari.\n2. Open `Plugins -> Chat Assistant`.\n3. Leave `Base URL` as `http://127.0.0.1:11434` unless your Ollama server is elsewhere.\n4. Choose a model from the `Model` dropdown or type a model tag manually.\n5. Use `Load` if you want to warm the selected model before the first request.\n6. Start chatting, or use the Library for repeatable tasks and reusable code.\n\nThe assistant works best when prompts describe a concrete action. If you already have Python code, paste it into the Prompt box and use `Run My Code`. If pasted code fails or needs adaptation to the current viewer, use `Refine My Code`.\n\n### Optional Voice Input\n\n`Advanced -> Voice Input` opens a small reusable voice window for users who want a local speech-to-text path inside napari.\n\nThe flow is:\n- choose the microphone input device\n- start and stop recording\n- watch the live input-level meter to confirm the mic is active\n- review and edit the transcript\n- press `Run` or hit `Enter` to execute it directly\n\nThis feature is intentionally optional and advanced. It depends on [`faster-whisper`](https://github.com/SYSTRAN/faster-whisper) being installed in the same Python environment as napari.\n\nExamples:\n- `Inspect the selected layer`\n- `Preview threshold on em_2d_snr_mid`\n- `Apply gaussian denoise to em_2d_snr_low with sigma 1.2`\n- `Fill holes in mask_messy_2d`\n- `Remove small objects from mask_messy_2d with min_size 64`\n- `Measure labels table for rgb_cells_2d_labels`\n- `Annotate template_blob_labels with publication-style callout labels`\n- `Create a max intensity projection from em_3d_snr_mid along axis 0`\n- `Crop em_2d_snr_high to the bounding box of em_2d_mask with padding 8`\n- `Extract ROI values from em_2d_snr_mid using em_2d_mask`\n- `Prepare this viewer for image review`\n- `Undo last workflow`\n\n## Core Workflow\n\n1. Open your image or volume in napari.\n2. Use `Layer Context` to inspect loaded layers or insert exact layer names into the Prompt box.\n3. Ask for a concrete action, or browse `Actions` when you want deterministic execution.\n4. Use `Templates` for reusable prompt/code examples and `Shortcuts` for repeated one-click workflows.\n5. Use generated code, `Run My Code`, or `Refine My Code` when a task needs custom Python.\n6. Save useful prompts, code snippets, workspace state, and shortcut layouts for later reuse.\n\nThis keeps inspection, analysis, code review, and workflow reuse close to the current napari session.\n\n## Synthetic Data Templates\n\nUse the Library `Templates > Code Templates > Data Setup` area or built-in code snippets to load repeatable synthetic datasets.\n\nCurrent built-in synthetic generators include:\n- Synthetic 2D SNR Sweep Gray\n- Synthetic 3D SNR Sweep Gray\n- Synthetic 2D SNR Sweep RGB\n- Synthetic 3D SNR Sweep RGB\n- messy masks 2D/3D\n\nThese create named layers so you can test built-in tools quickly without hunting for sample data. Labels layers from these synthetic datasets can also be used as ROIs for ROI inspection and value extraction.\n\n## Feature Summary\n\nBuilt-in workflows include:\n- layer inspection and live layer-context insertion\n- Quick Controls for fit view, zoom, 2D/3D mode, axes, scale bar, tooltips, overlays, grid view, and layer visibility\n- enhancement, thresholding, mask cleanup, connected components, labels measurement, projection, cropping, montage, and presentation helpers\n- ROI inspection and value extraction from `Labels`, `Shapes`, and line-based workflows\n- annotation overlays, including free text, particle labels, callouts, and boxed titles\n- workspace save/restore with a JSON manifest plus OME-Zarr assets for generated image and labels data\n- deterministic `Actions`, reusable `Templates`, and user-defined `Shortcuts`\n- optional advanced workflows such as Voice Input, SAM2, and Atlas Stitch\n\n### Code generation workflows\n\nWhen a request is not covered by a built-in tool, the assistant can return napari Python code instead of forcing the wrong tool.\n\nGenerated code can be:\n- copied to the clipboard\n- reviewed in chat\n- executed from the plugin\n- repaired or explained in place when you use `Refine My Code` on pasted or failed user code\n\nYou can also paste your own Python directly into the Prompt box and run it from the plugin with `Run My Code`, without switching to QtConsole.\n\nUse assistant-generated code when you want a reusable script or need custom logic beyond the current built-in tools.\n\nUse `Run My Code` when you already have Python you want to test quickly inside the current napari session.\n\nUse `Refine My Code` when your own code fails validation, errors at runtime, or needs adjustment to the current napari viewer state.\n\n## Library, Actions, And Shortcuts\n\nThe Library stores repeatable prompts, code snippets, templates, and deterministic actions.\n\nUseful behavior:\n- single click loads a prompt or code snippet into the editor\n- double click sends a prompt or runs a code snippet\n- templates can be previewed, loaded, or run\n- actions can be previewed, run, or added to `Shortcuts`\n- saved and recent prompts/code can be renamed, tagged, pinned, or cleared\n- shortcuts can be arranged, saved, loaded, and reused across sessions\n\nThis is now part of the core product direction: keep AI available, but let mature workflows become fast, button-driven, and reusable.\n\n## Optional Integrations\n\nIf `napari-nd2-spectral-ome-zarr` is installed, the assistant can open:\n- the ND2-to-OME-Zarr export widget\n- the Spectral Viewer widget\n- the Spectral Analysis widget\n\nThis lets chat act as an entry point for Nikon ND2 conversion and spectral workflows without rebuilding those UIs inside this plugin.\n\nInstall links:\n- GitHub: `https://github.com/wulinteousa2-hash/napari-nd2-spectral-ome-zarr`\n- napari Hub: `https://napari-hub.org/plugins/napari-nd2-spectral-ome-zarr.html`\n\n### Experimental SAM2\n\nBehavior:\n- SAM2 is accessed from `Advanced`, not from the main toolbar\n- `SAM2 Setup` is always available from `Advanced`\n- `SAM2 Live` stays disabled until the backend is configured and passes readiness checks\n- the rest of the assistant remains usable even if SAM2 is not configured\n\nCurrent setup expects:\n- a working Python environment that already includes the dependencies required by SAM2\n- an external SAM2 project path\n- a valid checkpoint path\n- a valid config path\n\n`napari-chat-assistant` now ships its own bundled SAM2 adapter in\n`napari_chat_assistant.integrations.sam2_adapter`, so users only need the SAM2 repo,\ncheckpoint, and config files in the normal places.\n\nThe `SAM2 Setup` dialog now includes:\n- `Auto Detect` to scan common local clone locations and fill likely project, checkpoint, and config paths\n- `Setup Help` for short setup commands and field tips\n\nMinimal install:\n\n```bash\ngit clone https://github.com/facebookresearch/sam2.git && cd sam2\npip install -e .\n```\n\nTypical setup flow:\n1. Start napari from the environment that contains your SAM2 dependencies.\n2. Open `Plugins -> Chat Assistant`.\n3. Open `Advanced -> SAM2 Setup`.\n4. Click `Auto Detect` first.\n5. Confirm or edit the SAM2 project path, checkpoint path, config path, and device.\n6. Click `Test`.\n7. Save the settings.\n8. Open `Advanced -> SAM2 Live` when the backend reports ready.\n\n## How It Works\n\nThe assistant is designed to operate within constrained napari workflows rather than as a general-purpose chatbot.\n\nThe current strategy is:\n1. collect structured napari viewer context\n2. build deterministic per-layer profile objects from the current viewer state\n3. add bounded approved session memory when available\n4. send that context and the user request to a local Ollama model\n5. the model returns a structured JSON response that specifies either:\n   - a normal reply\n   - a built-in tool call\n   - generated Python code\n6. run the selected registry-backed tool or expose the generated code through the UI\n7. update session memory from explicit user feedback or successful follow-up behavior\n\nThis keeps the assistant more grounded than a plain chat interface and makes common operations more reliable.\n\nCurrent viewer state and explicit user clarification remain the primary source of truth. Session memory is selective and secondary, not a full transcript memory system.\n\n## Recommended Models\n\nFor a broader list of models tested during development, see [docs/tested_models.md](docs/tested_models.md).\n\nGood starting choices:\n- `nemotron-cascade-2:30b`\n- `gpt-oss:120b`\n- `qwen3-coder-next:latest`\n- `qwen3.5`\n- `gemma4:e4b`\n\n`nemotron-cascade-2:30b` is the current default. Larger models may improve reasoning or code generation but require more RAM or VRAM; smaller tags are better for constrained systems.\n\n## Current Limitations\n\n- the dataset profiler is still Phase 1 and remains strongest on already-loaded napari layers rather than reader- or file-format-specific workflows\n- TIFF vs OME-Zarr adapter behavior is not implemented yet\n- ND2 and Zeiss reader-aware adapters are not implemented in this plugin\n- the tool registry is in progress; some tools are now registry-backed, but the migration is not complete yet\n- session memory is selective and bounded; it is not full conversation memory\n- model output can still be inconsistent, especially when falling back to generated code\n- some requests still miss built-in tools and fall through to code generation when a stronger built-in workflow would be preferable\n- generated code can still fail if the model invents incorrect napari APIs or unsupported imports\n- no image attachment or multimodal input pipeline yet\n- performance optimization for very large 2D/3D datasets is still in progress\n- hard native crashes in Qt/C-extension code may not be captured cleanly by the plugin crash log even when normal plugin errors are logged\n\nMost reliable current workflow:\n- use built-in tools for common layer inspection and mask/image actions\n- trust current viewer context and current layer profiles over any remembered prior turn\n- use the Library for repeated prompts, demo packs, and reusable code\n- use generated code when you want explicit review and control\n- use `Run My Code` when you already have working Python and want to test it directly inside napari\n\nFor demo and education workflows:\n- ask for code that uses the current napari `viewer`\n- avoid prompts that create a second `napari.Viewer()` or call `napari.run()`\n- prefer docked widgets over unmanaged popup windows for histogram or SNR teaching tools\n\n## Troubleshooting\n\n### Ollama not running\n\nIf `Test` fails after restarting your computer, Ollama is usually not running yet.\n\nStart it in a terminal:\n\n```bash\nollama serve\n```\n\nThen return to the plugin and click `Test` again.\n\n### Pulling a model\n\nModel downloads are intentionally handled outside the plugin.\n\nTo try a different model:\n- browse tags at `https://ollama.com/search`\n- type the tag into the plugin `Model` field if needed\n- pull it in a terminal, for example:\n\n```bash\nollama pull nemotron-cascade-2:30b\n```\n\nThen use `Test` to refresh the plugin state.\n\n### Logs and crash logs\n\nThe plugin writes two local log files:\n- `~/.napari-chat-assistant/assistant.log`\n- `~/.napari-chat-assistant/crash.log`\n\nUse these together with the terminal traceback when diagnosing crashes or unclear UI failures.\n\n### Local model telemetry\n\nIf telemetry is enabled, the plugin writes lightweight local model events to:\n- `~/.napari-chat-assistant/model_telemetry.jsonl`\n\nTelemetry is opt-in from `Session -> Telemetry` and includes summary, log, and reset controls for advanced users.\n\nGenerated code is also preflight-validated before execution for common dtype mistakes, unsupported napari imports, and unavailable `viewer.*` APIs. When validation blocks execution, the code remains visible and copyable for review or regeneration.\n\n## Release\n\nThis package is published to PyPI so napari Hub can discover it.\n\nFor maintainer release instructions and PyPI publishing setup, see [RELEASING.md](RELEASING.md).\n\n## Development\n\nBuild a release artifact:\n\n```bash\npython -m build\n```\n\n## License\n\nMIT.\n","description_content_type":"text/markdown","keywords":null,"home_page":null,"download_url":null,"author":"Wulin Teo","author_email":null,"maintainer":null,"maintainer_email":null,"license":null,"classifier":["Development Status :: 3 - Alpha","Framework :: napari","Operating System :: OS Independent","Programming Language :: Python","Programming Language :: Python :: 3","Programming Language :: Python :: 3.9","Programming Language :: Python :: 3.10","Programming Language :: Python :: 3.11","Programming Language :: Python :: 3.12","Programming Language :: Python :: 3.13","Topic :: Scientific/Engineering :: Image Processing"],"requires_dist":["matplotlib","markdown-it-py","napari","numpy","numcodecs","ome-zarr","qtpy","scipy","scikit-image","zarr","faster-whisper; extra == \"voice\"","pytest; extra == \"test\"","pytest-cov; extra == \"test\""],"requires_python":">=3.9","requires_external":null,"project_url":["Homepage, https://github.com/wulinteousa2-hash/napari-chat-assistant","Repository, https://github.com/wulinteousa2-hash/napari-chat-assistant","Issues, https://github.com/wulinteousa2-hash/napari-chat-assistant/issues"],"provides_extra":["voice","test"],"provides_dist":null,"obsoletes_dist":null},"npe1_shim":false}