High Performance JEE with Work Manager
and
Concise Asynch Beans - Work Manager
Saturday, March 24, 2012
Thursday, March 22, 2012
Kerberos Troubleshooting
Kerberos Troubleshooting
Resolving KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN
Use WireShark to determine the Kerberos Error
When problem determining Kerberos issues its a good idea to understand what is happening on the Network. Use Wire Shark to trace the network.
To quickly understand if Kerberos is failing identify the Kerberos Packets in the trace.
In the filter bar type Kerberos and click apply.
Only the KRB5 protocol packets will be displayed.
Analyze the values carefully. KRB5KDC_ERR_PREAUTH_REQUIRED is a valid error and not a problem. Pre-authentication is ok.
Are you seeing a KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN
Use nslookup to understand DNS configuration
nslookup can be used to understand the DNS configuration. nslookup does not use the local client cache or the the hosts file. Bear this in mind when testing. We are using windows here, which prints the dns server details first then the response.
nslookup myservice.mydomain.com
Server: mydnsserver.mydomain.com
Address: 192.168.1.2
name: myservice.mydomain.com
Address: 192.168.1.4
Here we can see the ip address for the service. Ok, now what? Well Kerberos will attempt to find out who is the real host for this service. How? It will use a reverse DNS lookup, which you can test with nslookup. Its important to do this more than once. Why? Because you want to know if the service suffers from round robin.
nslookup 192.168.1.4
Server: mydnsserver.mydomain.com
Address: 192.168.1.2
name: myservice.mydomain.com
Address: 192.168.1.4
Run this again:
nslookup 192.168.1.4
Server: mydnsserver.mydomain.com
Address: 192.168.1.2
name: myotherservice.mydomain.com
Address: 192.168.1.4
Oh dear, the dns address returned for this ip address just changed to myothersevice.mydomain.com. We only have an SPN defined for the myservice.mydomain.com.
Go back to your Wireshark Trace
Looking into the DNS server configuration we see that we have two A records for the 192.168.1.4 address from myservice.mydomain.com and myotherservice.mydomain.com. Furthermore we have two reverse DNS lookup PTR records. That doesn't look good!
Resolving this problem can be done by configuring a CNAME dns entry for the myotherservice.mydomain.com and removing the PTR record.
- To clear DNS name cache you type in: IPConfig /FlushDNS
- To clear NetBIOS name cache you type in: NBTStat –R
- To clear Kerberos tickets will need KList.exe: KList purge
- nslookup used to validate DNS - ignores cache and hosts file
- tracert used to validate DNS and Hosts file
- WireShark - your best freind
Resolving KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN
Use WireShark to determine the Kerberos Error
When problem determining Kerberos issues its a good idea to understand what is happening on the Network. Use Wire Shark to trace the network.
To quickly understand if Kerberos is failing identify the Kerberos Packets in the trace.
In the filter bar type Kerberos and click apply.
Only the KRB5 protocol packets will be displayed.
Analyze the values carefully. KRB5KDC_ERR_PREAUTH_REQUIRED is a valid error and not a problem. Pre-authentication is ok.
Are you seeing a KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN
Use nslookup to understand DNS configuration
nslookup can be used to understand the DNS configuration. nslookup does not use the local client cache or the the hosts file. Bear this in mind when testing. We are using windows here, which prints the dns server details first then the response.
nslookup myservice.mydomain.com
Server: mydnsserver.mydomain.com
Address: 192.168.1.2
name: myservice.mydomain.com
Address: 192.168.1.4
Here we can see the ip address for the service. Ok, now what? Well Kerberos will attempt to find out who is the real host for this service. How? It will use a reverse DNS lookup, which you can test with nslookup. Its important to do this more than once. Why? Because you want to know if the service suffers from round robin.
nslookup 192.168.1.4
Server: mydnsserver.mydomain.com
Address: 192.168.1.2
name: myservice.mydomain.com
Address: 192.168.1.4
Run this again:
nslookup 192.168.1.4
Server: mydnsserver.mydomain.com
Address: 192.168.1.2
name: myotherservice.mydomain.com
Address: 192.168.1.4
Oh dear, the dns address returned for this ip address just changed to myothersevice.mydomain.com. We only have an SPN defined for the myservice.mydomain.com.
Go back to your Wireshark Trace
Looking into the DNS server configuration we see that we have two A records for the 192.168.1.4 address from myservice.mydomain.com and myotherservice.mydomain.com. Furthermore we have two reverse DNS lookup PTR records. That doesn't look good!
Resolving this problem can be done by configuring a CNAME dns entry for the myotherservice.mydomain.com and removing the PTR record.
Wednesday, March 21, 2012
Tuesday, March 20, 2012
Linux Monitoring commands
free
free(1) displays the total amount of free and used physical memory and swap space in the system, as well as the buffers and cache consumed by the kernel.
Dstat
Dstat is a versatile replacement for vmstat, iostat, netstat and ifstat. Dstat overcomes some of their limitations and adds some extra features, more counters and flexibility. Dstat is handy for monitoring systems during performance tuning tests, benchmarks or troubleshooting.
Dstat allows you to view all of your system resources instantly, you can eg. compare disk usage in combination with interrupts from your IDE controller, or compare the network bandwidth numbers directly with the disk throughput (in the same interval).
Dstat gives you detailed selective information in columns and clearly indicates in what magnitude and unit the output is displayed. Less confusion, less mistakes.
iotop
Linux has always been able to show how much I/O was going on (the bi and bo columns of the vmstat 1 command). iotop is a Python program with a top like UI used to show of behalf of which process is the I/O going on.
sar and iostat
This package provides the sar and iostat commands for Linux. Sar and iostat enable system monitoring of disk, network, and other IO activity.
sdparm
SCSI disk parameters are held in mode pages. This utility lists or changes those parameters. Other SCSI devices (or devices that use the SCSI command set e.g. some SATA devices) such as CD/DVD and tape drives may also find parts of sdparm useful. Requires the linux kernel 2.4 series or later. In the 2.6 series any device node the understands a SCSI command set may be used (e.g. /dev/sda). In the 2.4 series SCSI device node may be used.
Fetches Vital Product Data pages. Can send commands to start or stop the media and load or unload removable media.
Warning: It is possible (but unlikely) to change SCSI disk settings such that the disk stops operating or is slowed down. Use with care.
mpstat
The mpstat command collects and displays performance statistics for all logical CPUs in the system. Users can define both, the number of times the statistics are displayed, and the interval at which the data is updated. When the mpstat command is invoked, it displays two sections of statistics. The first section displays the System Configuration, which is displayed when the command starts and whenever there is a change in the system configuration. The second section displays the Utilization Statistics which will be displayed in intervals and at any time the values of these metrics are deltas from pervious interval.
pidstat
The pidstat command is used for monitoring individual tasks currently being managed by the Linux kernel. It writes to standard output activities for every task selected with option -p or for every task managed by the Linux kernel if option -p ALL has been used. Not selecting any tasks is equivalent to specifying -p ALL but only active tasks (tasks with non-zero statistics values) will appear in the report.
top
The top program provides a dynamic real-time view of a running system. It can display system summary information as well as a list of tasks currently being managed by the Linux kernel. The types of system summary information shown and the types, order and size of information displayed for tasks are all user configurable and that configuration can be made persistent across restarts.
nmon
http://nmon.sourceforge.net/pmwiki.php
This systems administrator, tuner, benchmark tool gives you a huge amount of important performance information in one go. It can output the data in two ways
free(1) displays the total amount of free and used physical memory and swap space in the system, as well as the buffers and cache consumed by the kernel.
Dstat
Dstat is a versatile replacement for vmstat, iostat, netstat and ifstat. Dstat overcomes some of their limitations and adds some extra features, more counters and flexibility. Dstat is handy for monitoring systems during performance tuning tests, benchmarks or troubleshooting.
Dstat allows you to view all of your system resources instantly, you can eg. compare disk usage in combination with interrupts from your IDE controller, or compare the network bandwidth numbers directly with the disk throughput (in the same interval).
Dstat gives you detailed selective information in columns and clearly indicates in what magnitude and unit the output is displayed. Less confusion, less mistakes.
iotop
Linux has always been able to show how much I/O was going on (the bi and bo columns of the vmstat 1 command). iotop is a Python program with a top like UI used to show of behalf of which process is the I/O going on.
sar and iostat
This package provides the sar and iostat commands for Linux. Sar and iostat enable system monitoring of disk, network, and other IO activity.
sdparm
SCSI disk parameters are held in mode pages. This utility lists or changes those parameters. Other SCSI devices (or devices that use the SCSI command set e.g. some SATA devices) such as CD/DVD and tape drives may also find parts of sdparm useful. Requires the linux kernel 2.4 series or later. In the 2.6 series any device node the understands a SCSI command set may be used (e.g. /dev/sda). In the 2.4 series SCSI device node may be used.
Fetches Vital Product Data pages. Can send commands to start or stop the media and load or unload removable media.
Warning: It is possible (but unlikely) to change SCSI disk settings such that the disk stops operating or is slowed down. Use with care.
mpstat
The mpstat command collects and displays performance statistics for all logical CPUs in the system. Users can define both, the number of times the statistics are displayed, and the interval at which the data is updated. When the mpstat command is invoked, it displays two sections of statistics. The first section displays the System Configuration, which is displayed when the command starts and whenever there is a change in the system configuration. The second section displays the Utilization Statistics which will be displayed in intervals and at any time the values of these metrics are deltas from pervious interval.
pidstat
The pidstat command is used for monitoring individual tasks currently being managed by the Linux kernel. It writes to standard output activities for every task selected with option -p or for every task managed by the Linux kernel if option -p ALL has been used. Not selecting any tasks is equivalent to specifying -p ALL but only active tasks (tasks with non-zero statistics values) will appear in the report.
top
The top program provides a dynamic real-time view of a running system. It can display system summary information as well as a list of tasks currently being managed by the Linux kernel. The types of system summary information shown and the types, order and size of information displayed for tasks are all user configurable and that configuration can be made persistent across restarts.
nmon
http://nmon.sourceforge.net/pmwiki.php
This systems administrator, tuner, benchmark tool gives you a huge amount of important performance information in one go. It can output the data in two ways
- On screen (console, telnet, VNC, putty or X Windows) using
curses for low CPU impact which is updated once every two seconds. You
hit single characters on you keyboard to enable/disable the various
sorts of data.
- You can display the CPU, memory, network, disks (mini graphs or numbers), file systems, NFS, top processes, resources (Linux version & processors) and on Power micro-partition information.
- For lots of examples, see the "Screen shots" from the left menu.
- As you can see on the left lmon12e now in colour
- Save the data to a comma separated file for analysis and longer term data capture.
- Use this together with nmon Analyser Excel 2000 spreadsheet, which loads the nmon output file and automatically creates dozens of graphs ready for you to study or write performance reports.
- Filter this data, add it to a rrd database (using an excellent freely available utility called rrdtool). This graphs the data to .gif or .png files plus generates the webpage .html file and you can then put the graphs directly on a website automatically on AIX with no need of a Windows based machine.
- Directly put the data into a rrd database or other database for your own analysis
Saturday, March 10, 2012
IBM Support Assistant 5
IBM Support Assistant 5 has gone browser based!
After downloading and unzipping the zipped content, navigate to the zip directory and call ./start_isa.sh.
Once started, navigate in the browser to http://localhost:8080/isa5
After downloading and unzipping the zipped content, navigate to the zip directory and call ./start_isa.sh.
Once started, navigate in the browser to http://localhost:8080/isa5
JVM Troubleshooting
JVM Memory Troubleshooting
A good troubleshooting guide for JVM memory management
A good troubleshooting guide for JVM memory management
Subscribe to:
Posts (Atom)
