Seeds of Thoughts/ Facts / Knowledge: Knowledge

Showing posts with label Knowledge. Show all posts

Difference between "keyword" and "text" type in elastic search

Here we go with very simple example of statement "I'm running now!" and see how it will be splits into token and store into elastic search index.

statement : "I'm running now!"

if type="keyword" : [I'm running now!] - it will store in one token, you can not retrieve document it by matching 'running' or 'now' terms
if type="text": [i'm, running, now!] - it will store in three tokens, you can retrieve document by matching 'running' or 'now' terms

Text datatype

A field to index full-text values, such as the body of an email or the description of a product. These fields are analyzed, that is they are passed through an analyzer to convert the string into a list of individual terms before being indexed. The analysis process allows Elasticsearch to search for individual words within each full text field. Text fields are not used for sorting and seldom used for aggregations (although the significant text aggregation is a notable exception).
If you need to index structured content such as email addresses, hostnames, status codes, or tags, it is likely that you should rather use a keyword field.
Keyword datatype
A field to index structured content such as email addresses, hostnames, status codes, zip codes or tags.
They are typically used for filtering (Find me all blog posts where status is published), for sorting, and for aggregations. Keyword fields are only searchable by their exact value.
If you need to index full text content such as email bodies or product descriptions, it is likely that you should rather use a text field.

How do I find distinct values of date/ specific field based on date range in Elasticsearch

You need to do it this way, i.e. add the date range as a query to reduce the document set, and then run the terms aggregation only on the documents that fall into that date range:

Terms Aggregation

A multi-bucket value source based aggregation where buckets are dynamically built - one per unique value.

Date Range Aggregation

A range aggregation that is dedicated for date values. The main difference between this aggregation and the normal range aggregation is that the from and to values can be expressed in Date Math expressions, and it is also possible to specify a date format by which the from and to response fields will be returned. Note that this aggregation includes the from value and excludes the to value for each range.

POST index/_search?size=0
{
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "tstamp": {
              "gte": 1591795757000,
              "lte" : 1591890413000
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "result": {
      "terms": {
        "field": "tstamp",
        "size":171
      }
    }
  }
}

Issue with running a Update By query elasticsearch API in AWS

welcome to the knowledge gaps that I’ve been screaming about for many moons!

Elasticsearch - How to add new field to existing document by update by query. Elasticsearch: Back-fill old documents with email, documents would be around 1M+ records.

You can use the update by query API in order to add a new field to all your existing documents:

POST index/_update_by_query?conflicts=proceed&scroll_size=500

  "script": {

    "source": "ctx._source.email = 'rohitpatel0105@gmail.com'",

    "lang": "painless"

},

  "query": {

    "bool": {

      "must_not": [

          "exists": {

            "field": "email"

While running update by query API around 700k records have been updated and suddenly CPU utilization went to maximum state and nodes started to go down with cluster status "Red".

the most likely root cause is exactly that - my UBQ … which hit too many documents too fast and destroyed the cluster. in addition to UBQ being SO easy to over-match on and update docs you didn’t mean to.{ memory and gc graphs are towards the bottom of those dashboards }

I would suggest that you use the _tasks api to first find and annihilate any remaining, running UBQ tasks ( which will keep running independent of whether you are still connected ) … or just wait out the storm and see if the cluster ever comes back.
If deleting a problematic index isn't feasible, you can restore a snapshot, delete documents from the index, change the index settings, reduce the number of replicas, or delete other indices to free up disk space. The important step is to resolve the red cluster status before re-configuring your Amazon ES domain. Re-configuring a domain with a red cluster status can compound the problem and lead to the domain being stuck in a configuration state of Processing until you resolve the status.
Conclusion: Don't try to use update by query API with large docs without closing index directly in cluster.

Elasticsearch - How to add new field to existing document by update by query

Elasticsearch: Backfill old documents with email

You can use the update by query API in order to add a new field to all your existing documents:

POST index/_update_by_query

  "script": {

    "source": "ctx._source.email = 'rohitpatel0105@gmail.com'",

    "lang": "painless"

},

  "query": {

    "bool": {

      "must_not": [

          "exists": {

            "field": "email"

Have you discovered yourself in prevailing pandemic lockdown - Covid 19 in 2020? - By Hardik Patel

Every fade has a fall for sometime and so this growing pace too. It could take only 6 to 7 months for a tiny virus to grow in more than 125 countries. Does this call for a cause? Well its not a cause but seems to be a curse of how a man is a bundle of desires. We are not even a size of a virus against the nature and if we really have not understood these things, we are surely at a wrong race trust me.

Every luxury and lifestyle arrangements are affected but have you seen any impact on the water, air, fire and other forms of nature? Contrary they have purified but are we also purified on our thoughts? This uncertainties have brought lot of eye openers and those are points of discussions exactly on how we should learn things for not grooming and look decent to others but to curate ourselves as to how are we well prepared post this time to evaluate on knowing our souls and our heart which is loaded with the fake commitments and wrong requirements in life.

This time calls for a moment to think why do you be known to the world and why do you exist in this world? Is it because of the fulfillment's of other's dream or is it for the betterment of the society? Giving donations to poors or want to do something for poors? It can be so soothing and cool to be a part of the community by engaging ourselves on innovations and betterment of the community. Of course money will fall on you if these are done with the pure commitments. Falling money here I mean by is the karma's return and not money earned by doing something wrong.

Time is endless and it just ends for those who are done. You need to press hard the surface to extend the more seconds and minutes on life for regenarting and reopening up of our minds on how are we going to do something really different in this new normal.

We are not even a small hair in the entire galaxy, don't try to mess with nature, be natural on your thoughts and reinvent yourselves.

A small ending thought - cooking new dishes in this time is a waste of time if that is not helping the community.

We should spread equal love for community. We are the same as our color of the blood is same. Be wise and reinvent yourself.

- From a small piece of shit

H@rdik Patel.

How to create a war file directly inside jboss deployments folder(external folder) using maven in pom.xml

Kindly keep a configuration as below mentioned :

Pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0"

 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

 <modelVersion>4.0.0</modelVersion>

 <groupId>com.seedsofjava.blogspot.com</groupId>

 <artifactId>abc</artifactId>

 <packaging>war</packaging>

 <version>1.0</version>

 <name>abc</name>

 <build>

 <finalName>abc</finalName>

 <pluginManagement>

 <plugins>

 <plugin>

 <groupId>org.apache.maven.plugins</groupId>

 <artifactId>maven-compiler-plugin</artifactId>

 <version>3.5</version>

 <configuration>

 <source>${java-version}</source>

 <target>${java-version}</target>

 </configuration>

 </plugin>

 <plugin>

 <groupId>org.apache.maven.plugins</groupId>

 <artifactId>maven-war-plugin</artifactId>

 <configuration>

 <warName>abc</warName>

 <warSourceDirectory>src/main/webapp</warSourceDirectory>

 <failOnMissingWebXml>false</failOnMissingWebXml>

 <webResources>

 <resource>

 <directory>lib/</directory>

 <targetPath>WEB-INF/lib</targetPath>

 <includes>

 <include>**/*.jar</include>

 </includes>

 </resource>

 </webResources>

 <outputDirectory>C:/Softwares/jboss-eap-7.0/standalone/deployments</outputDirectory>

 </configuration>

 </plugin>

 </plugins>

 </pluginManagement>

 </build>

</project>

How to compare two string dates in hibernate using criteria if DB field

It is very simple and easy to do in hibernate using criteria condition. Let's say we have two fields like from date and to date As "String".

First we should consider below mentioned conditions while developing it

It will be like below cases:

User select fromdate and didnt select todate than 'it should fetch all records from fromdate to systemdat'
if user select only todate and not formdate than it should fetch all record till todate
If user enter both than it should filter the record in date range and display
If fromdate > todate then we should through error
Both are not given than it should find all the records till Systemdate i.e. Today

Code will be look like something as :

 Hibernate DAO layer:

if(!Utils.isNullOrEmpty(model.getDateFrom()) && !Utils.isNullOrEmpty(model.getDateTo())) { cr.add(Restrictions.sqlRestriction("TRUNC(COLUMN) >= TO_DATE('" + model.getDateFrom() + "','dd/mm/yyyy')"));

 cr.add(Restrictions.sqlRestriction("TRUNC(COLUMN) <= TO_DATE('" + model.getDateTo() + "','dd/mm/yyyy')"));

 else if(!Utils.isNullOrEmpty(model.getDateFrom()) && Utils.isNullOrEmpty(model.getDateTo())) {

 cr.add(Restrictions.sqlRestriction("TRUNC(COLUMN) >= TO_DATE('" + model.getDateFrom() + "','dd/mm/yyyy')"));

 //cr.add(Restrictions.sqlRestriction("TRUNC(COLUMN) <= TO_DATE('" + Utils.formatNewDate() + "','dd/mm/yyyy')"));

 else if(Utils.isNullOrEmpty(model.getDateFrom()) && !Utils.isNullOrEmpty(model.getDateTo())) {

 cr.add(Restrictions.sqlRestriction("TRUNC(COLUMN) <= TO_DATE('" + model.getDateTo() + "','dd/mm/yyyy')"));

Utils.java:

public class Utils {

 public static Boolean isNullOrEmpty(String str){

 return (str != null && !str.isEmpty()) ? false: true;

Best way to use Spring MVC ModelAttribute field in JSTL/JSP page in c:if

Let's say that you want to hide/show in GUI based on Spring MVC @ModelAttribute field, Kindly follow below mentioned steps :

Guess that you have one UserModel with field saveButton as string type for storing true/false.

UserModel.java


public class UserModel {
    private String saveButton = "true";
}

UserDetails.jsp

<c:if test="${userModel.saveButton == 'true'}">
	<input type="submit" class="btn btn-primary btn-sm" id="saveUserDetails" name="method" value="Save">
</c:if>

In this best way, we can achieve to hide/show button in spring mvc.

Seeds of Thoughts/ Facts / Knowledge

Search This Blog