logging - Logstash indexer is scalable? -

-

i using logstash shipper -> redis -> logstash indexer -> elastricsearch -> kibana

i understand redis , elastricsearch designed scalable. how logstash indexer, see on logstash main site show example of using single logstash indexer.

would enough handle logs? , suggestion on having many instances of logstash indexer?

if indexer bottleneck of such setup, add second logstash indexer on second machine same configuration, change redis source host , elasticsearch target host. when read same redis instance, use blocking reads ensure 1 redis entry read 1 indexer.

as components, @ horizontal scaling (more machines), indexer can scaled onto bigger machine.

if processing of single log events takes long, may try prevent work on indexer. e.g. done doing work on shipper side (using logstash filters) or having applications write json logs instead of "plain" loglines.


Comments

Post a Comment

Popular posts from this blog

python - Subclassed QStyledItemDelegate ignores Stylesheet -

-
i'm trying subclass qstyleditemdelegate remove focus rectangle in qcombobox . even though i'm calling base implementation of paint function , nothing else, result different. looks if parts of style-sheet influence bounding box of item taken consideration. class pstyleditemdelegate(qstyleditemdelegate): def __init__(self, *args, **kwds): super(pstyleditemdelegate, self).__init__(*args, **kwds) def paint(self, *args, **kwargs): qstyleditemdelegate.paint(*args, **kwargs) what have paint unmodified qstyleditemdelegate ? as suggested tried replacing pyside pyqt4 , works, appears bug. update pyside 1.1.2 1.2.1 result same. unfortunately switch breaks other parts of code, if there no other suggestions i'm going accept answer. edit bug tracked here
Read more

java - HttpClient 3.1 Connection pooling vs HttpClient 4.3.2 -

-
i have used httpclient3.1(java) connection pooling below : import org.apache.commons.httpclient.httpclient; import org.apache.commons.httpclient.multithreadedhttpconnectionmanager; import org.apache.commons.httpclient.params.httpclientparams; import org.apache.commons.httpclient.params.httpmethodparams; import org.apache.log4j.logger; public class httpclientmanager { private static logger logger = logger.getlogger(httpclientmanager.class); private static multithreadedhttpconnectionmanager connectionmanager = new multithreadedhttpconnectionmanager(); private static httpclientparams clientparams = new httpclientparams(); static { try { clientparams.makelenient(); clientparams.setauthenticationpreemptive(false); clientparams.setparameter(httpmethodparams.so_timeout, new integer(30000)); // set so_timeout clientparams.setparameter(httpmethodparams.head_body_check_timeout, new integer(3000)); // head responses
Read more

SQL: Divide the sum of values in one table with the count of rows in another -

-
is possible to, in single query, sum bunch of values in 1 table, , divide count of rows in another? there common id/key in both tables... fyi, win points least info provided in question. answer it's possible...here psuedo code since i'm guessing @ tables , columns select summing / counting whynot (select id, sum(whatever) summing where_ever ) inner join (select id, count(1) counting what_ever)b on a.id = b.id question = 'vague' group a.id
Read more