Log in

No account? Create an account
Agnus Dei's Journal

> recent entries
> calendar
> friends
> profile
> previous 20 entries

Wednesday, July 4th, 2018
1:15 am - Fun with Die Stats

Is it possible to be in love with a website?  I think I'm in love with this website -> http://rumkin.com/reference/dnd/diestats.php

Try things like:

A) Testing the new DnD advantage/disadvantage system where you roll 2 d20 and keep the higher if you have advantage, or roll 2d20 and keep the lower if you have disadvantage.

* Advantage =  type "2d20D1" (Average = 13.82)

* Neither = type "d20" (Average = 10.5)

* Disadvantage = type "2d20P1" (Average = 7.17)

B) Or test the RATM (roll and take middle) system where you roll 3 d20 and keep the middle value.

* RATM = type "3d20D1P1" (Average = 10.5 again, but with a much tighter standard deviation)

(comment on this)

Tuesday, July 3rd, 2018
11:35 pm - How to move a large mysql database using rsync (optimized for speed)
So I did something pretty cool Monday night (last night).

I have this database that's let say over a TB big. And let's say it normally takes 4-5 hours to copy it from one server to another using rsync.

So I came up with a way to run rsync(s) in parallel and maximize the throughput. It copied the whole thing in 35 minutes. So what used to take 4-5 hours to copy finished in 35 minutes using my new technique.

Step 1) . First command is for everything outside of the "big database" that you want to sync.  In this case I have only 5 running in parallel.  This should finish pretty quickly.

ls -1 /opt/mysql/ | egrep -v "^mybigdatabase$" | xargs -I {} -P 5 -n 1 rsync -rav --progress --inplace --no-whole-file /opt/mysql/{} myhostname.com:/opt/mysql/

Step 2) Now time to move the big database. The second command is for all the tables inside the big database you want to sync.  In this case I have 30 running in parallel (my host has 32 cores):

ls -1 /opt/mysql/mybigdatabase/ | xargs -I {} -P 30 -n 1 rsync -rav --progress --inplace --no-whole-file /opt/mysql/mybigdatabase/{} myhostname.com:/opt/mysql/mybigdatabase/

(comment on this)

Thursday, June 14th, 2018
10:31 am - perl examples

Keep this... like forever....


(comment on this)

Tuesday, May 29th, 2018
10:53 am - Desktop Icon for starting up VPN connection under Ubuntu
$ cat ~/Desktop/startvpn.desktop 
#!/usr/bin/env xdg-open
[Desktop Entry]
Name[en_US]=Start VPN
Name=Start VPN

$ cat /home/ballison/bin/start_vpn.sh
sudo openvpn --script-security 2 --config ~/work/vpn/current/vpn-my_account.ovpn 


Note - Created with: gnome-desktop-item-edit --create-new startvpn.desktop

(comment on this)

Thursday, May 24th, 2018
12:08 pm - Getting your DNS to work under OpenVPN on Ubuntu.


(comment on this)

Tuesday, May 22nd, 2018
3:52 pm - Fun with AWS, JSON and JQ

I have a lambda function that runs every night and makes AMI backups for any instance where we've set the Tag name "Backup" equal to "True".

So if Backup is not defined as a tag, the server gets backed up.  

So I wanted to know all the running instances where Backup was _not_ defined, so I could see what is _not_ being backed up.

Here's the command I came up with:

for i in us-east-1 us-west-2; do  
	aws ec2 describe-instances --filter "Name=instance-state-name,Values=running"  --output json --region $i \
		| jq '.Reservations[].Instances[] | select(contains({Tags: [{Key: "Backup"}, {Value: ""}]}) | not)' \
		| jq -r '.Tags[] | select(.Key=="Name") |.Value'; 
done  |sort 

View/Download: https://github.com/jackal242/brads_scripts/blob/master/aws_no_backup

(comment on this)

Monday, May 7th, 2018
1:03 pm - ssh - Unknown cipher type 'blowfish'

I just upgraded my Instance from Amazon Linux from "2016.03" to "2018.03" and now none of my autossh tunnels are working.

Turns out "blowfish" is no longer an accepable cypher for ssh.  Now it's "blowfish-cbc"

        [root@ip-10-0-0-89 init.d]# ssh -c blowfish 0
       Unknown cipher type 'blowfish'

         [root@ip-10-0-0-89 init.d]# ssh -V      
        OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017

They renamed the cypher to "blowfish-cbc".

         [root@ip-10-0-0-89 init.d]# ssh -c blowfish-cbc 0


(comment on this)

Monday, October 23rd, 2017
11:29 am - Wrote a script for get AWS IAM policies

# Description: Script to list all and resolve all policies associated with a given IAM user


(comment on this)

Monday, September 25th, 2017
12:47 am - bin/fix-strongvpn.sh

brad-allison-mbp:~ brad.allison$ cat bin/fix-strongvpn.sh 


sudo ifconfig en0 down; 

sleep 1; 

sudo route flush ; 

sleep 1; 

sudo ifconfig en0 up

(comment on this)

Friday, August 11th, 2017
2:45 pm - How to replace zgrep with pigz

When you are running zgrep you are basically actually running gzip with the decompression options piped to grep.

So to replace zgrep with pigz (to make it much much much faster), just pigz -dc the file piped to grep (which is basically what zgrep is doing).  Except as you know pigz will automatically thread out for the number of cores in your hosts to make it super fast.

Here's an example from one of my scripts.


  zgrep "$firm" /u/pound.log-20170${i}${y}.gz 


gzip -cdfq -- /u/pound.log-20170${i}${y}.gz |grep "$firm" 


pigz -dc /u/pound.log-20170${i}${y}.gz | grep "$firm" 

(comment on this)

Monday, June 5th, 2017
1:18 pm - How to audit the configuration of the DNS entry for DNS Failover in AWS Route 52.
How to audit the configuration of the DNS entry for DNS Failover in AWS Route 52.

$ aws route53 list-resource-record-sets --hosted-zone-id Z2I27QCOOT2SB2 --query "{ResourceRecordSets:ResourceRecordSets[?Name == 'myhostlala.mydomainfofo.com.'].{HealthCheckId:HealthCheckId}}"
    "ResourceRecordSets": [
            "HealthCheckId": "XXXXXXX-YYYY-4cdd-8400-XXXXXXXXX"
            "HealthCheckId": "XXXXXXX-YYYY-48c2-bb42-XXXXXXXXXX"

$ aws route53 get-health-check --health-check-id XXXXXXX-YYYY-4cdd-8400-XXXXXXXXX  --query HealthCheck.HealthCheckConfig.[{Type:Type},{Port:Port}]
        "Type": "TCP"
        "Port": 443

(comment on this)

Monday, March 20th, 2017
10:35 am - perl split example
Because I can never ever ever remember how to use split correctly in perl:
($count_of_worst_ip,$worst_ip,$culprit) = (split(/ /,$WORST_COUNT_PER_MINUTE_PER_IP ))[0,1,2];

(comment on this)

Tuesday, November 15th, 2016
7:36 pm - cool sed trick
Wrote something cool tonight.

I like this sed trick. I need to use it more. It basically uses the date stamp string to match that string and until the end of file. So you are only grepping from the first instance of that, until the end of file.

And the date string is set to now minus 10 minutes and then I remove the last character. So if it's currently "2016-11-16 00:18" then it looks from "2016-11-16 00:0" down until end of file.

So the window is always going to be the last 10-20 minutes of logs that it looks at.

ERROR_STRING="com.amazonaws.AmazonClientException: Unable to execute HTTP request: Timeout waiting for connection from pool"
DATE_REGEX=$(date "+%Y-%m-%d %H:%M" -d "10 min ago" | sed s'/.$//')  # Example 2016-11-16 00:0
COUNT=$(sed -n "/$DATE_REGEX/,\$p" ~tomcat/logs/mytomcat.log | grep "$ERROR_STRING" | wc -l )

if [ "$COUNT" -gt "0" ]; then

(comment on this)

Monday, November 14th, 2016
1:34 pm - A break down of S3 failed connections from tomcat broken down by count per hour.
A break down of S3 failed connections from tomcat broken down by count per hour.

awk 'BEGIN{RS="\n2016";ORS="\n\n2016";FS=OFS="\n"}/com.amazonaws.AmazonClientException: Unable to execute HTTP request: Timeout waiting for connection from pool/' ~tomcat/logs/mytomcat.log.2016-11-12 | grep -o '....-..-.. ..:' | sort | uniq -c
    406 2016-11-12 16:
    784 2016-11-12 17:
    379 2016-11-12 18:

(comment on this)

Wednesday, September 28th, 2016
3:35 pm - Convert mysql timestamp to human readable date
tail -10000 mysql-slow.log |awk -F'SET timestamp=' 'NF > 1{ system("date -d @" $2) }{print $_}' > foo.log

That will insert a line before the "SET timestamp=" line of the actual date for the query.

(comment on this)

3:34 pm - List the all the licenses installed on your Linux server

List the all the licenses you are using in your distribution of Linux and group them and count them, with the most used on top. :)

for i in `rpm -qa --queryformat '%{name}\n'`; do echo -n "$i:"; rpm -qi $i | grep ^License | head -1 ; echo ; done | awk 'NF' | cut -d: -f3 | sort | uniq -c | sort -rn

(comment on this)

Sunday, August 14th, 2016
8:29 pm - How to Disable SELinux (and remove the "dot" permissions)

Disabling SELinux

Check/Verify the SELinux current status

Run the following command:



[root@osboxes]# sestatus
SELinux status:                 enabled

Disable SELinux on the server

Run the following commands:

sed -i "/SELINUX/  s/enforcing/disabled/" /etc/selinux/config 

Confirm SELinux is disabled

[root@osboxes]# sestatus
SELinux status:                 disabled

View SELinux permissions

SELinux context remains associated with files regardless whether or not SELinux is enabled.

If you want to see the actual SELinux context associated with the files you can use either of following commands

ls -Z 
ls --lcontext


[root@osboxes rc.d]# cd /etc/rc.d/

[root@osboxes rc.d]# ls -alt
total 76
drwxr-xr-x. 121 root root 12288 Aug 15 00:41 ..
drwxr-xr-x.   2 root root  4096 Aug  6 07:33 rc0.d
drwxr-xr-x.   2 root root  4096 Aug  6 07:33 rc1.d
drwxr-xr-x.   2 root root  4096 Aug  6 07:33 rc2.d

[root@osboxes rc.d]# ls -Z 
drwxr-xr-x. root root system_u:object_r:etc_t:s0       init.d
-rwxr-xr-x. root root system_u:object_r:initrc_exec_t:s0 rc
drwxr-xr-x. root root system_u:object_r:etc_t:s0       rc0.d
drwxr-xr-x. root root system_u:object_r:etc_t:s0       rc1.d
drwxr-xr-x. root root system_u:object_r:etc_t:s0       rc2.d

[root@osboxes rc.d]# ls --lcontext 
total 60
drwxr-xr-x. 2 system_u:object_r:etc_t:s0       root root  4096 Aug  6 07:33 init.d
-rwxr-xr-x. 1 system_u:object_r:initrc_exec_t:s0 root root  2617 May 11 20:32 rc
drwxr-xr-x. 2 system_u:object_r:etc_t:s0       root root  4096 Aug  6 07:33 rc0.d
drwxr-xr-x. 2 system_u:object_r:etc_t:s0       root root  4096 Aug  6 07:33 rc1.d
drwxr-xr-x. 2 system_u:object_r:etc_t:s0       root root  4096 Aug  6 07:33 rc2.d

Remove SELinux permissions from files

SELinux context remains associated with files regardless whether or not SELinux is enabled.

If you are like me and you find the trailing "dot" annoying in the permissions, the following command will remove the dots (remove the SELinux context).

sudo setfattr -h -x security.selinux 


sudo setfattr -h -x security.selinux /home

find /home -print0 |xargs -0 -n 1 sudo setfattr -h -x security.selinux

find /home -exec sudo setfattr -h -x security.selinux {} \;

(comment on this)

Monday, August 8th, 2016
7:06 am - Plot Holes in Suicide Squad
All the things wrong with Suicide Squad

1- Dude blows himself up  - The soldier who Flag tells to detinate the bomb BLOWS HIMSELF UP.     If you order someone to kill themselves, do you really think they are going to just kill themselves.

2- Harley Throw gun to Deadshot who already has gun on his arm.

*- The Enchantress did not teleport.   Instead of teleporting away, The Enchantress makes an illusion of Deadshot's daughter to try and stop him.  She could have just teleported away.

*- Harley could I have just left, and would have.    At the end when amanda waller shows up with the app on her phone and tells them they are going back to prison, Harley's device was disabled.    So Harley could have just left.  There was nothing holding her there.

(comment on this)

Friday, August 5th, 2016
11:01 am - Apple Command + Enter = Middle Mouse Click in Linux
1- Install xdotool.

2- Add a keyboard shortcut for Apple Command + Enter = Middle Mouse Click in Linux

(comment on this)

Tuesday, August 2nd, 2016
1:03 pm - How to grep a Java Stacktrace

If you've ever had to deal with java stacktraces you know that they are a real pain to have to try and grep through.

If you want to just capture the stacktrace that matches a single string you are screwed because it's a multiple line glob.

Here's how to grep a java stacktrace using good old awk.  For this example, I'm using the date stamp (2016) for my delimiter for a new stacktrace stanza.

The string I'm searching for is "Status Code: 404" and notice how it returns just the stacktrace for me that matches that string.

[foo@myhost ~]# awk 'BEGIN{RS="\n2016-";ORS="\n\n2016-";FS=OFS="\n"}/Status Code: 404/' ~tomcat/logs/tomcat.log |more
08-02 16:10:37,581 http-8443-71 ERROR DefaultExceptionHandler:82 - Unhandled exception in exception handler. - XXXXXXXXXX
com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: XXXXXXXXXXX), S3 Extended Request ID: XXXXXXXXXXXXXXXX
    at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1160)
    at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:748)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:467)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:302)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
    at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1191)
    at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1059)
    at sun.reflect.GeneratedMethodAccessor1236.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at net.sourceforge.stripes.controller.DispatcherHelper$6.intercept(DispatcherHelper.java:467)
    at net.sourceforge.stripes.controller.ExecutionContext.proceed(ExecutionContext.java:158)
    at net.sourceforge.stripes.controller.ExecutionContext.proceed(ExecutionContext.java:155)
    at net.sourceforge.stripes.controller.ExecutionContext.proceed(ExecutionContext.java:155)
    at net.sourceforge.stripes.controller.ExecutionContext.proceed(ExecutionContext.java:155)
    at net.sourceforge.stripes.controller.BeforeAfterMethodInterceptor.intercept(BeforeAfterMethodInterceptor.java:113)
    at net.sourceforge.stripes.controller.ExecutionContext.proceed(ExecutionContext.java:155)
    at net.sourceforge.stripes.controller.ExecutionContext.wrap(ExecutionContext.java:74)
    at net.sourceforge.stripes.controller.DispatcherHelper.invokeEventHandler(DispatcherHelper.java:465)
    at net.sourceforge.stripes.controller.DispatcherServlet.invokeEventHandler(DispatcherServlet.java:278)
    at net.sourceforge.stripes.controller.DispatcherServlet.service(DispatcherServlet.java:160)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at net.sourceforge.stripes.controller.StripesFilter.doFilter(StripesFilter.java:247)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
    at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:436)
    at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:374)
    at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:302)
    at com.hazelcast.web.WebFilter$RequestWrapper$1.forward(WebFilter.java:428)
    at org.tuckey.web.filters.urlrewrite.NormalRewrittenUrl.doRewrite(NormalRewrittenUrl.java:213)
    at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:171)
    at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)
    at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)
    at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:381)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

(comment on this)

> previous 20 entries
> top of page