I would like to present a project I have been working on called nTask to distribute tasks across different computers:

nTask_small

What is nTask Link to heading

nTask is a versatile program that uses API communications and WebSocket to distribute tasks, whether they are commands or programs, among different computers. The main idea behind nTask is to allow clients to send task requests to a manager, which then assigns these tasks to available workers. The workers execute the tasks and send the results back to the manager, which then stores the data in a database. Additionally, nTask can also send the output to a designated URL or API for further integration with other programs.

In this blog post, I will walk you through the process of installing, configuring, and using nTask with a practical example. If you need more information, you can visit the nTask GitHub repository:

Readme Card

Installation and Configuration Link to heading

To get started with nTask, I recommend using my preferred method, which involves using Docker Compose for the server and database installation, and a manual installation on a VPS for the worker. This setup allows for greater flexibility when it comes to adding new tools or dependencies to nTask workers in the future.

Installing and Configuring the Manager Link to heading

To install the manager component of nTask, follow these steps:

  1. Install nTask:

    go install github.com/r4ulcl/nTask@latest

  2. Clone the nTask repository:

    git clone github.com/r4ulcl/nTask

  3. Generate a new SSL certificate by running the following command:

    cd nTask
bash generateCert.sh

    This will create the necessary SSL certificate files.

  4. Open the docker-compose.yml file and update the MySQL username and password.

version: '3.4'
services:
  manager:
    image: r4ulcl/ntask-manager
    #build:
    #  context: .
    #  dockerfile: ./manager/Dockerfile
    restart: unless-stopped
    container_name: nTask_manager
    ports:
      - 8080:8080
    depends_on:
      - db
    env_file: .env
    command: manager --swagger --verbose --debug 
    volumes:
      - ./manager.conf:/config/manager.conf
      - ./ssh.conf:/config/ssh.conf
      - ./output/:/config/output/
      - ./certs/:/config/certs/
      - ./ssh_key:/config/ssh_key
  db:
    image: mysql
    command: --default-authentication-plugin=caching_sha2_password
    restart: unless-stopped
    #ports:
    #  - 3306:3306
    environment:
      MYSQL_ROOT_PASSWORD: your_password_root
      MYSQL_USER: your_username
      MYSQL_PASSWORD: your_password
      MYSQL_DATABASE: manager
    volumes:
      - ./db:/var/lib/mysql

  1. Edit the manager.conf file and make the following changes:

    • Update the users section with the OAuth credentials for each user.
    • Update the workers section with the OAuth key for the workers. You can generate multiple OAuth keys for different workers if needed.
    • Modify the statusCheckSeconds value to specify the time interval (in seconds) between each check of the workers.
    • Adjust the StatusCheckDown value to define the time, in seconds, to wait before permanently removing a worker from the database if it doesn’t respond.
    • Update the dbUsername and dbPassword values to match the MySQL credentials specified in the docker-compose.yml file.
    • Optionally, set a diskPath to store the task execution information when it is completed.
{
  "users": {
    "user1" : "WLJ2xVQZ5TXVw4qEznZDnmEEV",
    "user2" : "WLJ2xVQZ5TXVw4qEznZDnmEE2",
    "user3" : "WLJ2xVQZ5TXVw4qEznZDnmEE3"
  },
    "workers": {
      "workers" : "IeH0vpYFz2Yol6RdLvYZz62TFMv5FF"
  },
  "statusCheckSeconds": 10,
  "StatusCheckDown": 360,
  "port": "8080",
  "dbUsername" : "your_username",
  "dbPassword" : "your_password",
  "dbHost" : "db",
  "dbPort" : "3306",
  "dbDatabase" : "manager",
  "diskPath": "./output",
  "certFolder": "./certs/manager/"
}

Installing and Configuring the Worker Link to heading

To install the worker component of nTask, follow these steps:

  1. Install Go on your client machine. You can use this script I created to automate the installation.

  2. Once Go is installed, run the following command to download and install nTask:

    go install github.com/r4ulcl/nTask@latest

  3. Copy the ca-cert.pem file from the manager folder.

  4. Create the worker.conf file and update the following information:

    • Set the number of iddleThreads to specify the number of threads to run in parallel on this worker.
    • Specify the IP address and port of the manager in the managerIP and managerPort fields.
    • Set the managerOauthToken to the OAuth token needed to connect to the manager.
    • Update the insecureModules and modules fields to define the custom modules you want to use in your tasks.
    • Set the CA config with the path to ca-cert.pem.
{
  "name": "",
  "iddleThreads": 2,
  "managerIP" : "nTask_manager",
  "managerPort" : "8080",
  "managerOauthToken": "IeH0vpYFz2Yol6RdLvYZz62TFMv5FF",
  "CA": "./certs/ca-cert.pem",
  "insecureModules": false,
  "modules": {
    "sleep": "/usr/bin/sleep",
    "curl": "/usr/bin/curl",
    "cat": "/usr/bin/cat",
    "echo": "/usr/bin/echo",
    "grep": "/usr/bin/grep",
    "rm": "/usr/bin/rm",
    "nmap": "nmap",
    "nmapIPs": "bash ./worker/modules/nmapIPs.sh",
    "touch": "/usr/bin/touch",
    "exec": ""
  }
}

Secure SSH Tunnel Configuration (Optional) Link to heading

For added security, you can use an SSH tunnel to access the manager without exposing the port to the internet. To do this, follow these steps:

  1. Create a ssh.conf file with the worker information using the following format:
{
  "ipPort": {
    "<IP1>" : "<PORT1>",
    "<IP2>" : "<PORT2>"
  },
  "sshUsername": "root",
  "privateKeyPath": "./ssh_key",
  "privateKeyPassword": ""
}

Replace <IP1>, <PORT1>, <IP2>, <PORT2>, etc. with the actual IP addresses and ports of the workers.

  1. Update the docker-compose.yml file to include the ssh.conf and ssh_key volumes:

    - ./ssh.conf:/config/ssh.conf
- ~/.ssh/ssh_key:/config/ssh_key

    This will ensure that the SSH configuration is shared with the Docker container.

    Note: Make sure to replace ~/.ssh/ssh_key with the actual path to your SSH private key file.

    With this configuration, the manager will establish an SSH connection with the workers and forward the local port 8080 for secure access without exposing it to the internet.

Usage examples Link to heading

Now that you have completed the installation and configuration steps, let’s dive into some usage examples using the scripts provided in the nTask repository.

Simple Example Link to heading

The first example uses the scriptExample.sh script, which shows how to use a bash script as a module. This script executes the nmapIPs module, which runs a bash script which execute an nmap command with the -sn option to scan for IPs and then applies a grep filter to extract only the relevant IPs.

In this script we have functions to send tasks by POST to the manager and functions to get tasks and wait for them to finish. First the variables are configured with the token, the nTask manager IP, etc.

# ... 
# Define vars
## oauth token, IP, port, range to scan
oauthToken="WLJ2xVQZ5TXVw4qEznZDnmEE2"
nTaskIP="127.0.0.1"
nTaskPort="8080"
scanRange="127.0.0.1/28"
url="https://$nTaskIP:$nTaskPort"
# Array to store task IDs
task_ids=()

Once this is done you run the nmapIPs module, which does an nmap ping only scan and returns the list of active IPs. And it waits for the task to run with its ID.

# Connect to Manager
## Send nmap ping only range
command="{\"module\": \"nmapIPs\", \"args\": \"$scanRange\"}"
task_data="{\"command\": [$command],\"priority\": 0}"
task_id=$(send_post_request "$url/task" "$oauthToken" "$task_data")
# add task_id to array
task_ids+=("$task_id")
echo task_ids: $task_ids

for task in "${task_ids[@]}"; do
    echo "$task"
done

# Wait for task done
OUTPUTS=($(wait_tasks $url $oauthToken "${task_ids[@]}"))
# ...

Once this is done the result of the task is displayed.

scriptExample-nmapIPs

As you can see in the output, the script successfully retrieves the list of IPs.

You can modify this script by changing the command in the following line (line 91):

command="{\"module\": \"curl\", \"args\": \"ipinfo.io\"}"

scriptExample-ipinfo

Advanced Example Link to heading

For a more advanced example, you can try the scriptExampleAdvanced.sh script. This script builds upon the previous example by scanning each IP obtained from the first task using nmap. This way, you can start with a range of IP addresses, filter out the up ones, and scan them using multiple workers.

So after getting the response from the first command we run the following to scan each IP separately.

# ... 
# For each IP execute nmap again but complete as a task
for IP in `echo -e $OUTPUT` ; do
    echo $IP
    command="{\"module\": \"nmap\", \"args\": \"$IP\"}"
    task_data="{\"command\": [$command],\"priority\": 0}"
    task_id=$(send_post_request "$url/task" "$oauthToken" "$task_data")
    echo task_ids: $task_id
    # add task_id to array
    task_ids+=("$task_id")
done

# Wait for task done
OUTPUTS=($(wait_tasks $url $oauthToken "${task_ids[@]}"))
# ...

Here we can see the output of the script:

scriptExampleAdvanced

Executing the Same Task in Multiple Workers Link to heading

In some scenarios, you may want to execute the same task in all or a specific list of workers. To demonstrate this functionality, you can use the scriptExample1command.sh script. This script uses curl to retrieve the public IP of each worker.

Basically the process is to get the workers that are connected and create a task for each of them.

# ... 
WORKERS=$(get_workers "$url/worker" "$oauthToken")

echo "$WORKERS" | jq


# Get all workers and send the same task to each
for row in $(echo "${WORKERS}" | jq -r '.[] | @base64'); do
    _jq() {
     echo ${row} | base64 --decode | jq -r ${1}
    }
    NAME=$(_jq '.name')

    command="{\"module\": \"curl\", \"args\": \"-s ipinfo.io/ip\"}"
    task_data="{\"command\": [$command],\"priority\": 0, \"workerName\": \"$NAME\"}"
    task_id=$(send_post_request "$url/task" "$oauthToken" "$task_data")
    echo task_id $task_id
    # add task_id to array
    task_ids+=("$task_id")
done

# Wait for task done
OUTPUTS=($(wait_tasks $url $oauthToken "${task_ids[@]}"))
# ...

Here we can see the output of the script:

scriptExample1command

Future Plans Link to heading

In the near future, I plan to integrate nTask with the DigitalOcean API to enable seamless connections and automatic management of workers. Stay tuned for updates!

Readme Card