[Django]-Why does Elasticsearch have a terrible performance indexing small amount of data inside docker?

3👍

I don’t remember how I resolved this problem almost a year ago, but I have some ideas that might help. There are several problems with that setup:

  1. Official instructions describe a multi-node cluster that consists of three nodes. For single-node cluster you should specify discovery.type=single-node. Single node cluster is suitable only for a development environment. For production, I suggest to leave docker and set up a multiple-server cluster with ansilbe.
  2. It’s better to use a recent version of elasticsearch
  3. There are too many shards

A good practice is to ensure the amount of shards for each node stays below 20 per GB of heap that is configured.

Check out this tutorial to find out more.

  1. Make sure that you have enough space on hard disk and don’t receive an error flood stage disk watermark [95%] exceeded on.

Here is my current setup for elasticsearch:

services:
 
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10.0
    container_name: es01
    environment:
      - node.name=es01_local
      - cluster.name=es_cluster_local
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms1024m -Xmx1024m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - esnet

Command docker-compose docker-compose.yml exec es01 curl -XGET http://localhost:9200/_cluster/health?pretty=true returns:

{
  "cluster_name" : "es_cluster_local",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 6,
  "active_shards" : 6,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 6,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 50.0
}

Leave a comment