_source
fieldedit
The _source
field contains the original JSON document body that was passed
at index time. The _source
field itself is not indexed (and thus is not
searchable), but it is stored so that it can be returned when executing
fetch requests, like get or search.
If disk usage is important to you then have a look at
synthetic _source
which shrinks disk usage at the cost of
only supporting a subset of mappings and slower fetches or (not recommended)
disabling the _source
field which also shrinks disk
usage but disables many features.
Synthetic _source
edit
Synthetic _source
is Generally Available only for TSDB indices
(indices that have index.mode
set to time_series
). For other indices
synthetic _source
is in technical preview. Features in technical preview may
be changed or removed in a future release. Elastic will apply best effort to fix
any issues, but features in technical preview are not subject to the support SLA
of official GA features.
Though very handy to have around, the source field takes up a significant amount
of space on disk. Instead of storing source documents on disk exactly as you
send them, Elasticsearch can reconstruct source content on the fly upon retrieval.
Enable this by setting mode: synthetic
in _source
:
PUT idx { "mappings": { "_source": { "mode": "synthetic" } } }
While this on the fly reconstruction is generally slower than saving the source documents verbatim and loading them at query time, it saves a lot of storage space.
Synthetic _source
restrictionsedit
There are a couple of restrictions to be aware of:
-
When you retrieve synthetic
_source
content it undergoes minor modifications compared to the original JSON. -
The
params._source
is unavailable in scripts. Instead use thedoc
API or thefield
API. -
Runtime fields without a script, and runtime
fields that access
_source
are currently not supported for indices that use synthetic_source
. Use a scripted runtime field that accesses fields using doc values or thefield
API instead. -
Synthetic
_source
can be used with indices that contain only these field types:
Synthetic _source
modificationsedit
When synthetic _source
is enabled, retrieved documents undergo some
modifications compared to the original JSON.
Arrays moved to leaf fieldsedit
Synthetic _source
arrays are moved to leaves. For example:
PUT idx/_doc/1 { "foo": [ { "bar": 1 }, { "bar": 2 } ] }
Will become:
{ "foo": { "bar": [1, 2] } }
This can cause some arrays to vanish:
PUT idx/_doc/1 { "foo": [ { "bar": 1 }, { "baz": 2 } ] }
Will become:
{ "foo": { "bar": 1, "baz": 2 } }
Fields named as they are mappededit
Synthetic source names fields as they are named in the mapping. When used
with dynamic mapping, fields with dots (.
) in their names are, by
default, interpreted as multiple objects, while dots in field names are
preserved within objects that have subobjects
disabled. For example:
PUT idx/_doc/1 { "foo.bar.baz": 1 }
Will become:
{ "foo": { "bar": { "baz": 1 } } }
Alphabetical sortingedit
Synthetic _source
fields are sorted alphabetically. The
JSON RFC defines objects as
"an unordered collection of zero or more name/value pairs" so applications
shouldn’t care but without synthetic _source
the original ordering is
preserved and some applications may, counter to the spec, do something with
that ordering.
Disabling the _source
fieldedit
Though very handy to have around, the source field does incur storage overhead within the index. For this reason, it can be disabled as follows:
PUT my-index-000001 { "mappings": { "_source": { "enabled": false } } }
Think before disabling the _source
field
Users often disable the _source
field without thinking about the
consequences, and then live to regret it. If the _source
field isn’t
available then a number of features are not supported:
-
The
update
,update_by_query
, andreindex
APIs. - On the fly highlighting.
- The ability to reindex from one Elasticsearch index to another, either to change mappings or analysis, or to upgrade an index to a new major version.
- The ability to debug queries or aggregations by viewing the original document used at index time.
- Potentially in the future, the ability to repair index corruption automatically.
If disk space is a concern, rather increase the
compression level instead of disabling the _source
.
Including / Excluding fields from _source
edit
An expert-only feature is the ability to prune the contents of the _source
field after the document has been indexed, but before the _source
field is
stored.
Removing fields from the _source
has similar downsides to disabling
_source
, especially the fact that you cannot reindex documents from one
Elasticsearch index to another. Consider using
source filtering instead.
The includes
/excludes
parameters (which also accept wildcards) can be used
as follows:
PUT logs { "mappings": { "_source": { "includes": [ "*.count", "meta.*" ], "excludes": [ "meta.description", "meta.other.*" ] } } } PUT logs/_doc/1 { "requests": { "count": 10, "foo": "bar" }, "meta": { "name": "Some metric", "description": "Some metric description", "other": { "foo": "one", "baz": "two" } } } GET logs/_search { "query": { "match": { "meta.other.foo": "one" } } }