KodeKloud 100 Days of MLOps – Day 2: Fix a Broken JupyterLab Server Configuration

Step-by-step walkthrough of Day 2 task in KodeKloud 100 Days of MLOps challenge: diagnosing configuration errors, enabling root access, setting correct IP bindings, and launching a JupyterLab server. .

MLOpsKodeKloudPythonVirtual EnvironmentDevOps

Misconfigured infrastructure is one of the most common blockers in MLOps workflows. When team configurations prevent core services from launching, understanding how to inspect, diagnose, and repair configuration files is essential.

In Day 2 of the KodeKloud 100 Days of MLOps challenge, we tackle a broken JupyterLab Server Configuration for the xFusionCorp Industries data science team, resolving network binding issues, directory paths, and root privilege blocks.

Before get in to the task and sollution I would like to leave a note about where and how a Jupyter server configs can be broken.

When troubleshooting or setting up a JupyterLab / Jupyter Notebook server, configuration failures typically fall into 6 different reasons.

Common Jupyter Server Configuration Pitfalls (Corrected & Expanded)

1. Network & IP Bindings (c.ServerApp.ip, c.ServerApp.port)

These settings determine where Jupyter listens for incoming connections.

Binding to 127.0.0.1 or a specific IP instead of 0.0.0.0

c.ServerApp.ip = "127.0.0.1"

This only allows connections from the local machine. Inside Docker or Kubernetes, Jupyter runs in its own network namespace. A reverse proxy (Nginx, Traefik, Apache) or Docker port mapping cannot reach it.

As a result you will most likely get:

  • 502 Bad Gateway
  • Connection Refused
  • Browser never connects

Correct configuration

c.ServerApp.ip = "0.0.0.0"

0.0.0.0 means:

Listen on every available network interface.

This is almost always required for:

  • Docker
  • Kubernetes
  • Cloud VMs
  • Reverse proxies

For a purely local laptop installation where only you access Jupyter, 127.0.0.1 is perfectly fine and even slightly more secure.

Port Conflicts

Setting c.ServerApp.port to a port already used by another service, or mismatched with the host port mapping

Example:

c.ServerApp.port = 8888

If another application already uses port 8888:

Error: Address already in use

In containers another mistake is mismatch of what proxy expects and what containor expose.

Container

Jupyter -> 8888

Docker

-p 8000:8888

Nginx

proxy_pass http://container:8000;

The proxy expects 8000 while the container exposes 8888.

port_retries

f c.ServerApp.port_retries = 0 is set and the designated port is busy, the server crashes immediately instead of searching for the next open port.

Default config is as below.

c.ServerApp.port_retries = 50

If the designated port 8888 is busy it will automatically search and try other free ports like ,

8889
8890
8891
...

If configured as

c.ServerApp.port_retries = 0

the server immediately exits if the selected port is occupied.

This is useful in production because you generally want the service to fail rather than unexpectedly switch ports.

2. Authentication, Permissions & Security

Running as root

Jupyter intentionally refuses to run as root for security reasons. If c.ServerApp.allow_root = True (or —allow-root) is missing in a container running as root, the startup process immediately aborts. If you looked at my terminal in bottom of this post you’ll see the actual error massage I got from Kodekloud task. But Container images often run as root.

Example error:

Running as root is not recommended.
Use --allow-root to bypass.

Required configuration:

c.ServerApp.allow_root = True

or

jupyter lab --allow-root

In the task i used config file instead of bash because task was about the config file. And something I learned about this is the Better practice is running the container as a non-root user whenever possible.

Token & Cross-Site Request Forgery (XSRF) Mismatches is another thing to address lets break it down

Authentication Tokens

Default:

c.ServerApp.token = "<random-token>"

or in newer versions:

c.IdentityProvider.token = "<random-token>"

This protects access to the notebook.

This is not common but some developers disable it:

c.IdentityProvider.token = ""

This is acceptable only if another authentication layer already exists (for example OAuth, JupyterHub, corporate SSO, or a trusted reverse proxy). Exposing an unauthenticated Jupyter server to the internet is a serious security risk.

XSRF Protection

Jupyter protects against Cross-Site Request Forgery.

Default:

c.ServerApp.disable_check_xsrf = False

Normally in production this should remain enabled.

Problems appear mostly in proxied environments (e.g., Nginx, Kubernetes Ingress, or Cloudflare):

  • reverse proxies rewrite headers incorrectly (Host, X-XSRFToken, Origin).
  • cookies are stripped
  • cross-origin requests are misconfigured

When XSRF checks fail, the UI opens, but functional operations break with HTTP 403 Forbidden.

  • Save fails
  • Execute fails
  • WebSocket authentication problems
  • HTTP 403 errors

To fix this you can Disable XSRF:

c.ServerApp.disable_check_xsrf = True

Disabling XSRF protection exposes your server to malicious web pages executing code on your instance if an authenticated user visits them.should only be used for trusted internal deployments or debugging. It should not be the first fix for browser issues.Fixing header and proxy configurations on your ingress layer is always the preferred long-term solution.

Filesystem Permissions

When Jupyter starts, it needs read and write access to its runtime directory (by default, ~/.local/share/jupyter/runtime/) to store essential session state and security files:

  • jpserver-<pid>.json: Contains connection details (port, URL, token) for active server instances.
  • jpserver-<pid>-open.html: The auto-generated launcher file used to open the UI in a browser.
  • notebook_cookie_secret (or jupyter_cookie_secret): The secret key used to sign browser session cookies for security.

Permission Failures (PermissionError)

If Jupyter lacks write permissions to this directory, it will throw a PermissionError or fail to initialize cookie secrets during boot.

Common Causes:

  • Docker Mounts: Volumes mounted into containers owned by root while running the container under a non-root user.
  • Read-Only Filesystems: Immutable container roots or restricted sandbox environments.
  • Mismatched $HOME: Running sudo without sudo -E or having $HOME pointing to a directory the current user doesn’t own.
  • Missing Runtime Path: The directory structure does not exist on disk.

The Fix

Ensure the directory exists and grant the running user full read/write access:

# Create the runtime directory path
mkdir -p ~/.local/share/jupyter/runtime

# Restore ownership and permissions for the current user
chown -R $USER:$USER ~/.local/share/jupyter
chmod -R u+rw ~/.local/share/jupyter

Why adding chown helps:

In Docker environments (especially when switching between root and non-root container users), chmod alone can fail if the directory is strictly owned by root. Including chown -R $USER:$USER makes the fix complete across all edge cases!

3. Working Directory (root_dir)

This determines what users see inside Jupyter.

Example:

c.ServerApp.root_dir = "/workspace/notebooks"

Non-Existent Workspace Directory

If the configured root notebook directory (e.g., /workspace/notebooks) does not exist on disk, Jupyter will fail during the initialization phase.

Typical Error Output:

FileNotFoundError: [Errno 2] No such file or directory: '/workspace/notebooks'
traitlets.traitlets.TraitError: The 'root_dir' trait of a ServerApp instance expected a directory, but '/workspace/notebooks' does not exist.

Unlike some frameworks, Jupyter does not auto-create missing notebook directories on startup. If the target path isn’t present, initialization halts immediately.

The Fix Always ensure the target directory exists on disk prior to launching the server:

mkdir -p /workspace/notebooks

Relative vs. Absolute Paths

Avoid setting relative paths for working directories because the resulting root directory will depend entirely on where you execute the jupyter launch command in your terminal.

Avoid Relative Configurations

c.ServerApp.root_dir = "./notebooks"

If you use a relative path like ./notebooks, launching the server from different directories produces different working spaces:

# Serves /home/user/notebooks
cd /home/user
jupyter lab

# Serves /tmp/notebooks
cd /tmp
jupyter lab
- Instead:

Always Use Absolute Paths Specify explicit, absolute paths in your configuration files to ensure consistent behavior across all environments, startup scripts, and systemd/Docker services:

c.ServerApp.root_dir = "/home/user/notebooks"
  1. Configuration Hierarchy, Traitlets, and Syntax Pitfalls

Jupyter relies on Traitlets a Python based configuration and type checking system to manage all server parameters. Understanding how Traitlets evaluate classes, handle syntax errors, and apply configuration precedence will save you hours of debugging.

1. Class Hierarchy: ServerApp vs. NotebookApp

Modern JupyterLab (v3/v4) is built on top of jupyter_server, which uses the ServerApp class for configuration. Older versions (Notebook v6 and earlier) used the legacy NotebookApp class.

Modern (Jupyter Server / JupyterLab 3+):

c.ServerApp.port = 8888
c.ServerApp.ip = '0.0.0.0'
c.ServerApp.root_dir = '/root/notebooks'

Legacy (Notebook v6 and earlier):

c.NotebookApp.port = 8888
c.NotebookApp.ip = '0.0.0.0'
c.NotebookApp.notebook_dir = '/root/notebooks'

While Jupyter Server provides temporary aliases for many legacy NotebookApp traits via notebook_shim, relying on NotebookApp in modern setups is discouraged as these shims are slated for removal in future major releases. Always prefer c.ServerApp.

2. Configuration Files Are Executable Python

Because jupyter_lab_config.py is an actual Python file, any syntax or runtime error will prevent the interpreter from loading the file, crashing the server before it even initializes.

Common Failure Modes: Unclosed Strings (SyntaxError):

c.ServerApp.ip = "0.0.0.0

Result: The Python parser fails on startup, throwing a SyntaxError: EOL while scanning string literal.

Undefined Functions/Variables (NameError):

# Accidentally calling an unimported or non-existent helper
setup_custom_hooks()

Result: Python throws a NameError: name ‘setup_custom_hooks’ is not defined, aborting the boot process completely.

3. Configuration Precedence Order

Jupyter loads configuration files from multiple filesystem levels in a strict hierarchy. Settings loaded from higher-priority levels override lower-priority settings.

Load Order (Lowest to Highest Priority):


1. System-wide Configs (/etc/jupyter/)
       │
       ▼
2. Virtual Environment Configs ($VIRTUAL_ENV/etc/jupyter/)
       │
       ▼
3. User-Level Configs (~/.jupyter/)
       │
       ▼
4. Explicit Config File (--config /path/to/custom_config.py)
       │
       ▼
5. Direct CLI Flags (--port=8888 --ip=0.0.0.0)

If a setting defined in ~/.jupyter/jupyter_lab_config.py isn’t applying, check if a CLI flag or explicit —config flag passed to the jupyter lab executable is overriding it.

Debugging Config Search Paths To view the exact directories Jupyter searches for configuration files on your specific system, run:

jupyter --paths

Example Output:

config:
    /root/.jupyter
    /root/code/ml-env/etc/jupyter
    /usr/local/etc/jupyter
    /etc/jupyter

5. Virtual Environments & Kernels

Wrong Executable Path

A common issue in Python environments occurs when packages are installed into a virtual environment (e.g., ml-env/), but the system-wide binary is launched instead.

  • Incorrect: /usr/bin/jupyter lab (uses system Python)
  • Correct: /root/code/ml-env/bin/jupyter lab (uses virtual environment Python)

How to identify:

  • ModuleNotFoundError or ImportError when trying to import libraries that are installed inside your virtual environment.
  • Mismatched package versions between terminal and notebook sessions.

Verification

Check which executable is being invoked:

which jupyter

To guarantee execution from the intended virtual environment, invoke the binary using its absolute path:

/path/to/venv/bin/jupyter lab --config /path/to/jupyter_lab_config.py

Broken Kernels & Dead Environments

Jupyter manages execution environments via kernel specification files (kernel.json).

Example kernel.json:

{
  "argv": [
    "/opt/myenv/bin/python",
    "-m",
    "ipykernel_launcher",
    "-f",
    "{connection_file}"
  ],
  "display_name": "Python 3 (myenv)",
  "language": "python"
}

If the underlying virtual environment directory (/opt/myenv) is moved, renamed, or deleted, the JupyterLab server will still boot normally, but running any notebook using that kernel will fail.

Typical Error Signals

  • UI Notification: Kernel failed to start or Kernel Died
  • Terminal Log: FileNotFoundError: [Errno 2] No such file or directory: '/opt/myenv/bin/python'

Fixing Broken Kernels

List all registered kernels and their disk locations:

jupyter kernelspec list

Remove or re-install the invalid kernel spec:

# Remove the invalid spec
jupyter kernelspec uninstall <kernel_name>

# Register a new kernel spec from inside the active virtual environment
/path/to/venv/bin/python -m ipykernel install --user --name myenv --display-name "Python (myenv)"

6. Reverse Proxies & WebSockets

JupyterLab relies heavily on two HTTP communication mechanisms:

  • Standard REST HTTP requests for static assets, tree navigation, and REST APIs.
  • WebSockets (ws:// / wss://) for kernel execution, terminal emulation, and real-time state sync.

If a reverse proxy (like Nginx, Apache, or Traefik) is placed in front of Jupyter without proper WebSocket upgrade support, the static UI will load, but code execution will fail.

Client Browser ──> Nginx (Reverse Proxy) ──[ HTTP + WebSockets ]──> Jupyter Server

How to identify:

  • JupyterLab UI loads normally, but the status indicator stuck on “Connecting…” or “Disconnected”.
  • Opening a terminal inside JupyterLab results in a blank, un-interactive screen.

Required Nginx Configuration

Ensure the Nginx configuration includes explicit HTTP/1.1 upgrade directives:

location / {
    proxy_pass http://127.0.0.1:8888;

    # Force HTTP/1.1 for WebSocket support
    proxy_http_version 1.1;

    # Pass Upgrade headers for WebSockets
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";

    # Preserve host and client context
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}

Serving Under a Base Path (base_url)

When hosting JupyterLab behind a subpath (e.g., https://example.com/jupyter/), update c.ServerApp.base_url in your configuration file so asset URLs and API requests resolve correctly:

# jupyter_lab_config.py
c.ServerApp.base_url = '/jupyter/'

Without this setting, static assets (.js, .css) and API endpoints will return 404 Not Found errors when accessed through the proxy subpath.

7. TLS / HTTPS Offloading

In most production environments, SSL/TLS termination is handled by an ingress controller or reverse proxy (e.g., Nginx, Cloudflare):

Internet ──[ HTTPS ]──> Nginx ──[ Unencrypted HTTP ]──> Jupyter Server (0.0.0.0:8888)

In this setup, Jupyter runs over HTTP internally and does not require TLS certificates configured in its python configuration file.

Direct TLS Configuration (No Proxy)

If Jupyter handles TLS termination directly without a reverse proxy, specify the certificate paths in jupyter_lab_config.py:

c.ServerApp.certfile = '/etc/ssl/certs/jupyter.crt'
c.ServerApp.keyfile = '/etc/ssl/private/jupyter.key'

Common Failure: If the specified certfile or keyfile paths do not exist or lack read permissions for the running user, JupyterLab will fail on startup.

8. Extension Compatibility Diagnostics

Incompatible or outdated extensions can cause server startup failures or prevent the frontend from rendering properly.

Inspecting Installed Extensions

To check active server extensions:

jupyter server extension list

To check active frontend extensions:

jupyter labextension list

Troubleshooting Extension Errors

If an extension causes crash loops or crashes the server, disable or uninstall it:

# Disable a server extension without uninstalling
jupyter server extension disable <extension_name>

# Uninstall a lab extension
jupyter labextension uninstall <extension_name>

9. Logging & Debugging

If the JupyterLab server fails to start without printing a detailed error message, run it in debug mode to inspect full startup traces, configuration evaluation, and extension loading:

CLI Debug Flag

/path/to/venv/bin/jupyter lab --config /path/to/config.py --debug

Configuration-based Debug Flag

# jupyter_lab_config.py
c.Application.log_level = 'DEBUG'

Key Output Areas in Debug Logs:

  • Config Searching: Shows which configuration files were discovered and loaded.
  • Extension Initialization: Pinpoints which server extension threw an unhandled exception.
  • Binding Events: Confirms the exact IP interface and port the server is listening on.

Production Best Practices

SettingRecommended
ip0.0.0.0 (containers), 127.0.0.1 (local only)
portFixed port with proxy mapping
port_retries0 in production, default locally
allow_rootAvoid if possible; use only in containers when necessary
tokenKeep enabled unless protected by external authentication
disable_check_xsrfKeep False unless you fully understand the implications
root_dirUse an absolute, existing directory
ConfigurationPrefer ServerApp settings over legacy NotebookApp
Reverse ProxyEnable HTTP/1.1 and WebSocket upgrade headers
Virtual EnvironmentLaunch the jupyter executable from the correct environment
LoggingUse --debug when troubleshooting

Summary

Most Jupyter configuration issues fall into one of these categories:

  1. Incorrect network binding (127.0.0.1 vs 0.0.0.0)
  2. Port conflicts or incorrect proxy mapping
  3. Authentication or XSRF misconfiguration
  4. Filesystem permission problems
  5. Invalid or non-existent root_dir
  6. Mixing legacy NotebookApp and modern ServerApp settings
  7. Wrong Python environment or broken kernels
  8. Missing WebSocket support in reverse proxies
  9. Extension compatibility issues
  10. TLS/HTTPS or base_url misconfiguration

Understanding how these layers interact makes diagnosing Jupyter deployment issues much faster, whether running locally, in Docker, Kubernetes, or behind a production reverse proxy.

Task Scenario & Requirements

A teammate previously configured a JupyterLab server inside a Python virtual environment located at /root/code/ml-env/. However, the server was failing to function properly due to incorrect network settings and path configurations.

Here were the strict requirements for the final running state:

  1. Bind to all network interfaces (0.0.0.0).
  2. Listen on port 8888.
  3. Set the root notebook directory to /root/notebooks/ (ensuring the directory exists on disk).
  4. Enable root access since the server runs in a containerized environment as root.

🔍 Step 1: Diagnosing the Misconfigurations

The initial team overrides in /root/code/jupyter_lab_config.py looked like this:

# --- xFusionCorp team overrides (review before starting the server) ---
c.IdentityProvider.token = ''
c.ServerApp.disable_check_xsrf = True
c.ServerApp.root_dir = '/root/wrong-path'
c.ServerApp.port = 8000
c.ServerApp.ip = '1.1.1.1'

What was broken?

  • Invalid IP Binding (1.1.1.1): Restricting the server to an external IP stopped local and reverse-proxy connections (like Nginx) from reaching Jupyter.
  • Wrong Port (8000): The application stack expected traffic on 8888.
  • Non-existent Directory (/root/wrong-path): The target directory didn’t exist, and the team required /root/notebooks/.
  • Root Execution Blocked: Starting Jupyter as root without explicit permission caused a critical server crash, resulting in a 502 Bad Gateway from Nginx.

⚙️ Step 2: Applying the Fixes

1. Create the Notebooks Directory

First, make sure the target directory exists on disk:

mkdir -p /root/notebooks

2. Update jupyter_lab_config.py

Modify /root/code/jupyter_lab_config.py to fix the bindings, set the path, and explicitly allow execution as root:

# --- xFusionCorp team overrides ---
c.IdentityProvider.token = ''
c.ServerApp.disable_check_xsrf = True
c.ServerApp.root_dir = '/root/notebooks/'
c.ServerApp.port = 8888
c.ServerApp.ip = '0.0.0.0'
c.ServerApp.allow_root = True

🚀 Step 3: Starting and Verifying the Server

Launch the JupyterLab instance using the dedicated virtual environment binary:

/root/code/ml-env/bin/jupyter lab --config /root/code/jupyter_lab_config.py

Terminal Output Verification

[I 2026-07-27 06:03:31.122 ServerApp] Serving notebooks from local directory: /root/notebooks
[I 2026-07-27 06:03:31.122 ServerApp] Jupyter Server 2.20.0 is running at:
[I 2026-07-27 06:03:31.122 ServerApp] http://0.0.0.0:8888/lab
[I 2026-07-27 06:03:31.122 ServerApp]     http://127.0.0.1:8888/lab
[I 2026-07-27 06:03:50.254 ServerApp] 302 GET / (@10.244.189.210) 0.43ms

Notice the successful HTTP 302 GET / request at the end — the reverse proxy connected successfully, resolving the 502 Bad Gateway error and routing straight to the interactive workspace!

💡 Key Takeaways

  1. 0.0.0.0 vs Specific IPs: Always bind to 0.0.0.0 inside containers or VMs when you want external reverse proxies (like Nginx) or host machines to access the service.
  2. Container Security Flags: Jupyter intentionally guards against running as root. Adding c.ServerApp.allow_root = True in your config file avoids execution flags in startup scripts.
  3. Environment Isolation: Always call binaries directly from your target virtual environment (e.g., /root/code/ml-env/bin/jupyter) to prevent system-wide package pollution.
All Kodekloud 100 Days of MLOps posts