Service Deployment
Deployment has two stages: prepare a Kubernetes cluster, then deploy SQLRec and its dependencies. Minikube and existing clusters use the same download_resource.sh and deploy_components.sh; STORAGE_MODE selects how client files are stored.
System Requirements
The deployment machine can run AMD64/ARM64 Linux or Apple Silicon macOS. The first deployment needs access to image registries, Helm repositories, and resource download URLs.
| Deployment | Preparation |
|---|---|
| Minikube development environment | Linux uses the Docker driver; macOS uses vfkit, vmnet-shared networking, and VirtioFS mounts. deploy_minikube.sh installs missing tools and starts the cluster |
| Existing Kubernetes cluster | Configure kubeconfig and install kubectl, helm, curl, tar, gzip, envsubst, psql, and either sha256sum or shasum; Minikube is not required |
Minikube deployment on macOS requires macOS 14 or later and Homebrew. The main packages installed by the script are equivalent to:
brew install minikube vfkit docker docker-buildx helm gettext libpqThe script also installs kubernetes-cli if kubectl is missing and configures Buildx and vmnet-helper. Docker Desktop is not required on macOS. Use the checks under deploy/ for exact tool version requirements.
Quick Deployment (Minikube)
git clone https://github.com/sqlrec/sqlrec.git
cd sqlrec/deploy
# Install tools, start the cluster, and configure its default StorageClass
bash ./deploy_minikube.sh
kubectl get nodes -o wide
# Download clients/JARs, add Helm repositories, and install required operators
bash ./download_resource.sh
# Deploy SQLRec and its dependencies
bash ./deploy_components.sh
kubectl get pods --all-namespaces
# Verify the connection after required components are ready
cd ..
bash ./bin/beeline.shA successful SHOW TABLES; in Beeline confirms that the core SQLRec service is ready.
Minikube configuration:
- The script mounts the host's
datadirectory into Minikube at the same path. Clients and JARs use the host files directly, without uploading a second copy to PVCs. - On macOS, defaults are the host's physical core count, 80% of total memory, and a 256GB disk. Override them with
MINIKUBE_CPUS,MINIKUBE_MEMORY_PERCENT,MINIKUBE_MEMORY, orMINIKUBE_DISK_SIZE; an explicitMINIKUBE_MEMORYtakes precedence. Actual resource needs depend on enabled components and data volume. - Dynamic volumes use the project-installed
local-pathStorageClass by default, with data at/data/local-path-provisioneron the node. SetLOCAL_PATH_PROVISIONER_DATA_DIRbefore startup to use another absolute path. These dynamic volumes are separate from the hostPath PVs mounting host client directories. - NodePorts use the Minikube node address. Access through the host's physical IP from other LAN machines is not guaranteed.
Save Image Caches Manually
After deploying components, optionally run:
# Run from deploy; save workload images that have already been pulled
bash ./cache_images.sh saveCaches are stored in data/image-cache/<arch>. Later runs of deploy_minikube.sh load existing caches automatically. deploy_components.sh prints the save command but does not save images automatically. Image caches do not contain database data; deleting the Minikube cluster loses data in node dynamic volumes.
Deploy to an Existing Kubernetes Cluster
This flow installs components in the cluster selected by the current kubeconfig context. It does not create a cluster.
Prerequisites
- Ensure the deployment account can create namespaces, install operators/CRDs, and create the RBAC resources used by the scripts. Besides downloading files,
download_resource.shinstalls operators including CloudNativePG and Flink and configures Helm repositories. - Prepare a StorageClass supporting ReadWriteMany (RWX) and set
STORAGE_CLASSso clients and JARs can be shared across nodes. The script checks that the class exists; the cluster administrator must confirm RWX support. - Ensure a usable default StorageClass is available for data volumes such as PostgreSQL and RustFS. It can differ from the RWX class used for clients.
- Select a worker node address reachable from both the deployment machine and pods as
NODE_IP, and ensure the required NodePorts are accessible. Discovery selects a Ready worker that is not cordoned, excludes control-plane nodes, and uses its InternalIP. Set a reachable address explicitly if the deployment machine cannot reach that InternalIP. - If the deployment machine and cluster nodes have different architectures, set
CONTAINER_ARCH=amd64orarm64before downloading resources to select container Java and Hadoop native libraries. This does not change host Java or automatically reconcile different architectures within a cluster.
Deployment Example
Run these commands in the same terminal, replacing the example context, address, and StorageClass:
cd /path/to/sqlrec
kubectl config use-context your-cluster-context
kubectl get nodes -o wide
kubectl get storageclass
export NAMESPACE=sqlrec
export NODE_IP=10.0.0.10
export STORAGE_MODE=shared
export STORAGE_CLASS=nfs-rwx
export CONTAINER_ARCH=amd64
bash ./deploy/download_resource.sh
bash ./deploy/deploy_components.sh
kubectl get pods -n "${NAMESPACE}"
kubectl get pvc -n "${NAMESPACE}"
bash ./bin/beeline.shK8S_APISERVER_ADDR is read from the current kubeconfig by default, in the form k8s://https://.... It accesses the Kubernetes API and is separate from NODE_IP, which is used for NodePorts.
deploy_components.sh deploys PostgreSQL, RustFS/JuiceFS, Hadoop configuration, HDFS, Spark configuration, HMS, Flink SQL Gateway, SQLRec, and Kafka, Redis, and Milvus by default. Milvus runs in ${NAMESPACE}-milvus. MongoDB, Kyuubi, Jupyter, and observability components use their own deployment scripts. The default filesystem remains jfs://myjfs; access HDFS explicitly through hdfs://myhdfs.
HDFS uses one NameNode and one DataNode on the selected NODE_IP node. The deployment preserves existing metadata. Use separate capacity and availability planning for workloads that need more than this development setup.
Storage and Resource Synchronization
Storage Modes
| Item | hostpath (Minikube) | shared (existing cluster) |
|---|---|---|
| Template | deploy/storage/hostpath_pvc.yaml | deploy/storage/shared_pvc.yaml |
| Resources | Static hostPath PVs + PVCs, ReadWriteOnce | PVCs provisioned by the specified StorageClass, ReadWriteMany |
| File source | Host directories mounted into Minikube | Deployment machine files uploaded to shared PVCs |
| Upload behavior | Copy local configuration only; skip upload | A temporary pod mounts the PVCs and uploads changed files |
If STORAGE_MODE is unset, a context named minikube selects hostpath; all other contexts select shared. deploy_minikube.sh itself always uses hostpath. Remote clusters should use shared: local deployment directories do not automatically exist on remote nodes.
| Default PVC name | Contents | Capacity setting |
|---|---|---|
sqlrec-lib-pvc | Dependency JARs | LIB_STORAGE_SIZE, default 128Gi |
sqlrec-client-pvc | Hadoop, Hive, Spark, container Java clients and configuration | CLIENT_STORAGE_SIZE, default 128Gi |
Override PVC names with LIB_PVC_NAME and CLIENT_PVC_NAME. Existing PVC storage classes and access modes cannot be switched in place; the script stops on a mismatch. Use new PVC names and plan file migration when changing storage backends. hostPath PVs use the Retain reclaim policy; override their names with LIB_PV_NAME and CLIENT_PV_NAME.
RustFS provides S3 storage in standalone mode, and the deployment creates the buckets required by JuiceFS and Milvus. Data volumes always use the cluster's default StorageClass; STORAGE_CLASS for clients/JARs does not affect RustFS. Set capacities with RUSTFS_DATA_STORAGE_SIZE and RUSTFS_LOG_STORAGE_SIZE, defaulting to 128Gi and 1Gi respectively.
Resource Synchronization
Deployment scripts prepare clients, JARs, and configuration automatically. No manual PV/PVC template application or separate HMS configuration step is needed. After updating prepared resources, run bash ./deploy/storage/sync.sh from the repository root. Synchronization does not remove extra remote files; do not run it concurrently in one namespace.
Local resources are under deploy/data by default, or ${BASE_DIR}/data when configured: conf/ contains generated configuration, lib/ contains JARs, and client/ contains clients and Java. Shared-mode synchronization uploads runtime files and skips unchanged ones.
bash ./deploy/storage/sync.shProduction Configuration
The existing-cluster flow can be reused, but SRE teams should review component capacity, persistent storage, networking, credentials, and permissions for production. Default deployments include single-instance services and NodePorts; adapt them to your operational requirements.
Dependencies
| Service | Purpose |
|---|---|
| Kubernetes | Manage model training, export, and serving |
| PostgreSQL | Model, service, and function metadata |
| Hive Metastore | Hive table metadata |
| Flink SQL Gateway | Optional; executes forwarded Flink SQL/RPCs |
| Distributed storage | Model files and training data; RustFS + JuiceFS by default |
Use Kafka, Redis, Milvus, Spark, Kyuubi, and Jupyter according to workload needs. Kafka, Redis, and Milvus are already included in the default component deployment flow.
Script Parameters and Container Configuration
Scripts read deploy/env.sh. Override supported parameters using matching environment variables before execution, for example:
export NAMESPACE=dev
export SQLREC_VERSION=your-version
bash ./deploy/download_resource.sh
bash ./deploy/deploy_components.shKeep namespace, version, path, and architecture settings consistent between resource download and deployment. Main script parameters and container settings include:
| Parameter | Purpose |
|---|---|
NODE_IP | NodePort address reachable from both the deployment machine and pods |
SQLREC_POSTGRESQL_USER / SQLREC_POSTGRESQL_PASSWORD | SQLRec metadata database credentials |
HMS_POSTGRESQL_USER / HMS_POSTGRESQL_PASSWORD | HMS metadata database credentials |
SQLREC_POSTGRESQL_PORT / HMS_POSTGRESQL_PORT | Database NodePorts |
HMS_PORT | HMS NodePort |
FLINK_SQL_GATEWAY_ADDRESS | Gateway host; defaults to NODE_IP during deployment, and an empty address disables forwarding |
FLINK_SQL_GATEWAY_PORT | Gateway NodePort and SQLRec connection port, default 30018 |
FLINK_SQL_GATEWAY_CONNECT_TIMEOUT | SQLRec container runtime setting; connection and RPC read timeout, default 600000 ms (10 minutes) |
SQLREC_THRIFT_PORT / SQLREC_REST_PORT | JDBC/Beeline / REST NodePorts, default 30000 / 30001 |
RUSTFS_ACCESS_KEY / RUSTFS_SECRET_KEY | Shared S3 credentials |
DEPLOY_TIMEOUT | Deployment wait timeout, default 3600 seconds |
The SQLRec container's META_DB_URL and HIVE_METASTORE_URI are generated in deploy/sqlrec/sqlrec.yaml from NODE_IP and the corresponding ports. MODEL_BASE_PATH is fixed to /user/sqlrec/models in that template.
When starting SQLRec directly, set META_DB_URL and HIVE_METASTORE_URI for remote metadata. FLINK_SQL_GATEWAY_ADDRESS and TRACE_ENDPOINT default to empty and disable those features. For SQL-file mode, set SQL_SCHEMA_DIR instead of metadata endpoints.
Connect using CLI, Beeline, or Python. Source development and integration tests are covered in Development and Testing.
To use dependencies managed by SRE teams, adjust the corresponding deployment steps and YAML connection settings. The scripts do not skip component deployment when an external address is set. Exporting META_DB_URL or HIVE_METASTORE_URI alone does not replace the template addresses. Even running deploy/sqlrec/deploy.sh alone deploys PostgreSQL first.
SQLRec and Spark scripts create their ServiceAccounts and grant cluster-wide edit access through ClusterRoleBindings. For production, scope permissions to the required resources and namespaces using restricted Roles/RoleBindings.
Execution Variables and Result Parameters
These options are read only from the server process environment and default to enabled. SQL SET and request params cannot override them.
| Environment variable | Default | Description |
|---|---|---|
ENABLE_VARIABLE_ENV_FALLBACK | true | When an execution variable is absent (null), SQL reads the same-named environment variable; explicit empty strings do not trigger fallback |
EXECUTE_RESULT_INCLUDE_PARAMS | true | Include execution variables as params in SQL and function REST API results; disabling omits this JSON field without affecting variable use during execution |
Set either environment variable to false and restart the service to disable it. Fallback values are never written into execution variables: get_variables() and response params contain only explicit execution variables. Removing an explicit variable restores environment fallback on subsequent reads. With fallback enabled, SQL can read server process environment settings by variable name.
Entry Authentication
Direct server startup disables authentication by default. Deployment scripts enable it with development credentials sqlrec / abc123456. Override them before deployment:
| Environment variable | Server default | Description |
|---|---|---|
AUTH_ENABLED | false | Enable HTTP Basic and JDBC/Thrift SASL PLAIN |
AUTH_USERNAME | Empty | Comma-separated usernames |
AUTH_PASSWORD | Empty | Passwords paired with usernames by position |
AUTH_ENABLED=true AUTH_USERNAME=alice AUTH_PASSWORD='replace-with-your-password' \
bash deploy/sqlrec/deploy.shFor multiple accounts, set AUTH_USERNAME=alice,bob and AUTH_PASSWORD=alice-secret,bob-secret. Both lists must have the same length; entries must be non-blank, usernames unique, and usernames must not contain colons. Commas separate entries and cannot be escaped. Spaces are part of credentials; control characters are rejected. Invalid credentials prevent startup.
Set these values in the container environment or in a file loaded by bin/server.sh through SQLREC_ENV_CONF. Changes require a restart or redeployment; SQL SET and request params cannot override them. Deployment templates write credentials into container environment variables, including the rendered YAML.
- HTTP and UI require Basic credentials; missing or incorrect credentials return
401.GET /metricsremains anonymous. - Authenticated clients have the same permissions; role-based and per-resource authorization are not available.
- Basic and SASL PLAIN do not encrypt transport. Use HTTPS or an appropriate TLS tunnel where required.
Client examples, including Beeline and PyHive, are in Client Connections.
Gateway Forwarding Configuration
Deployment scripts install Flink SQL Gateway and configure SQLRec's connection address by default. Gateway serves requests that require Flink execution; local SQL, metadata queries such as SHOW CREATE TABLE and DESCRIBE, JDBC metadata, and persistent table/database/UDF metadata DDL do not depend on it.
Default settings require no changes. For an external Gateway, set the address and port listed above to its HiveServer2/Thrift endpoint. An empty address followed by redeployment disables forwarding; this does not skip default Flink component deployment or remove existing components.
export FLINK_SQL_GATEWAY_ADDRESS=""
bash ./deploy/sqlrec/deploy.shSee Architecture for execution boundaries and the SQL Reference for SET/RESET behavior.
Local SQL-File Metadata
| Environment variable | Default | Description |
|---|---|---|
SQL_SCHEMA_DIR | Empty string | Recursively load SQL definitions from this directory when nonempty; otherwise use remote metadata |
SQL_SCHEMA_DDL_ENABLED | false | Enable in-memory database, table, Java UDF, SQL function, and API DDL in file mode only |
Configure these in the container environment or SQLREC_ENV_CONF, then restart the service. Session SET cannot change the option. Changes are shared by sessions in one process, never rewrite SQL files, and are lost on restart. Separate instances do not share changes. FLUSH preserves in-memory changes. Model and Service lifecycle commands retain their existing restrictions. See Metadata DDL for supported statements.
SQL Initialization
deploy/sqlrec/deploy.sh waits for PostgreSQL, runs deploy/sql/master.sql, then deploys SQLRec. No manual SQL import is needed. Existing tables are reused; this file does not migrate existing table schemas.
If PostgreSQL is not ready, adjust PSQL_MAX_ATTEMPTS (default 30), PSQL_RETRY_INTERVAL (5 seconds), or PGCONNECT_TIMEOUT (10 seconds). SQL initialization errors stop deployment and roll back that initialization transaction; fix the reported SQL error before retrying.
Verification and Troubleshooting
Run from the repository root. If NAMESPACE is unset, these commands use the default sqlrec:
kubectl get pods -n "${NAMESPACE:-sqlrec}"
kubectl get pvc -n "${NAMESPACE:-sqlrec}"
bash ./bin/beeline.sh| Symptom | Check |
|---|---|
| Missing or invalid client/JAR | Run deploy/download_resource.sh first; if the error persists, remove the affected corrupt file or extracted directory and rerun |
| Shared PVC or sync pod remains Pending | Use kubectl describe pvc <name> -n <namespace>; check RWX support, capacity, and permissions. WaitForFirstConsumer classes begin binding after the sync pod is created |
| PostgreSQL connection retries exhausted | Check database pod readiness, NODE_IP, NodePorts, routes, and credentials. Fix SQL errors using the psql output |
| Sync pod already exists | Wait for the active sync to finish; remove a leftover pod only after confirming that the previous sync has stopped |
See Observability for metrics, traces, and logs after deployment.